[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:21:41 dist-test-slave-69rg krb5kdc[4989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:21:41 dist-test-slave-69rg krb5kdc[4989](info): set up 2 sockets
Aug 07 02:21:41 dist-test-slave-69rg krb5kdc[4989](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260807 02:21:42.950641 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.006s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:21:43 dist-test-slave-69rg krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:21:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:21:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:21:43.094658 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:40245
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:40245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:43.221071 5003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:43.221462 5003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:43.221554 5003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:43.225972 5003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:21:43.226068 5003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:43.226104 5003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:43.226145 5003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:21:43.226179 5003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:21:43.232532 5003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:40245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:40245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:43.234295 5003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:43.235821 5003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:43.250633 5011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:43.250746 5008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:43.252079 5003 server_base.cc:1034] running on GCE node
W20260807 02:21:43.252959 5009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:43.253873 5003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:43.255647 5003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:43.256871 5003 hybrid_clock.cc:648] HybridClock initialized: now 1786069303256890 us; error 70 us; skew 500 ppm
Aug 07 02:21:43 dist-test-slave-69rg krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:43.260340 5003 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:21:43.261725 5003 webserver.cc:460] Webserver started at http://127.4.221.126:32847/ using document root <none> and password file <none>
I20260807 02:21:43.262739 5003 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:43.262849 5003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:43.263293 5003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:43.266345 5003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "0c55efb174ac4320b6a7794e95ec14fb"
format_stamp: "Formatted at 2026-08-07 02:21:43 on dist-test-slave-69rg"
server_key: "75bb20189d66e40a42b28ce780d290ab"
server_key_iv: "0815242436b5b1652d24d07083f625ce"
server_key_version: "encryptionkey@0"
I20260807 02:21:43.267268 5003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "0c55efb174ac4320b6a7794e95ec14fb"
format_stamp: "Formatted at 2026-08-07 02:21:43 on dist-test-slave-69rg"
server_key: "75bb20189d66e40a42b28ce780d290ab"
server_key_iv: "0815242436b5b1652d24d07083f625ce"
server_key_version: "encryptionkey@0"
I20260807 02:21:43.272867 5003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:21:43.276834 5018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:43.278657 5003 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 02:21:43.278891 5003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "0c55efb174ac4320b6a7794e95ec14fb"
format_stamp: "Formatted at 2026-08-07 02:21:43 on dist-test-slave-69rg"
server_key: "75bb20189d66e40a42b28ce780d290ab"
server_key_iv: "0815242436b5b1652d24d07083f625ce"
server_key_version: "encryptionkey@0"
I20260807 02:21:43.279047 5003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:43.295849 5003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:43.300395 5003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:43.300705 5003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:43.316671 5003 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:40245
I20260807 02:21:43.316705 5070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:40245 every 8 connection(s)
I20260807 02:21:43.318248 5003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:21:43.323648 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 5003
I20260807 02:21:43.323796 5071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:43.323858 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:21:43.325506 4981 external_mini_cluster.cc:1428] Setting key 5f910a32b74cce206898a6cdaaf8ba81
I20260807 02:21:43.334281 5071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb: Bootstrap starting.
I20260807 02:21:43.339529 5071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:43.341728 5071 log.cc:824] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb: Log is configured to *not* fsync() on all Append() calls
I20260807 02:21:43.345293 5071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb: No bootstrap required, opened a new log
I20260807 02:21:43.352371 5071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c55efb174ac4320b6a7794e95ec14fb" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40245 } }
I20260807 02:21:43.352715 5071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:43.352813 5071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c55efb174ac4320b6a7794e95ec14fb, State: Initialized, Role: FOLLOWER
I20260807 02:21:43.353757 5071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c55efb174ac4320b6a7794e95ec14fb" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40245 } }
I20260807 02:21:43.354038 5071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Aug 07 02:21:43 dist-test-slave-69rg krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:43.354171 5071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:21:43.354329 5071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:43.355935 5071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c55efb174ac4320b6a7794e95ec14fb" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40245 } }
I20260807 02:21:43.356707 5071 leader_election.cc:302] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c55efb174ac4320b6a7794e95ec14fb; no voters:
I20260807 02:21:43.357164 5071 leader_election.cc:288] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:21:43.357453 5076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:21:43.358439 5076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [term 1 LEADER]: Becoming Leader. State: Replica: 0c55efb174ac4320b6a7794e95ec14fb, State: Running, Role: LEADER
I20260807 02:21:43.358909 5076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c55efb174ac4320b6a7794e95ec14fb" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40245 } }
I20260807 02:21:43.359076 5071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:21:43.362391 5077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c55efb174ac4320b6a7794e95ec14fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c55efb174ac4320b6a7794e95ec14fb" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40245 } } }
I20260807 02:21:43.362558 5077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [sys.catalog]: This master's current role is: LEADER
I20260807 02:21:43.363070 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:43.330346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.4.221.126:40245)
0807 02:21:43.331093 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:43.331139 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:43.335092 (+ 3953us) server_negotiation.cc:407] Connection header received
0807 02:21:43.340158 (+ 5066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:43.340187 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:43.340868 (+ 681us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:43.341412 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:43.344464 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:43.346776 (+ 2312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:43.348598 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:43.349463 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:43.349631 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:43.355084 (+ 5453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:43.355131 (+ 47us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:43.355149 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:43.355194 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:43.357601 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:43.358292 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:43.358298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:43.358303 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:43.358464 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:43.358951 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:43.358953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:43.358955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:43.359678 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:43.359887 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:43.361061 (+ 1174us) server_negotiation.cc:299] Negotiation successful
0807 02:21:43.361450 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":224,"threads_started":1}
I20260807 02:21:43.363227 5090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:21:43.363219 5078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c55efb174ac4320b6a7794e95ec14fb. Latest consensus state: current_term: 1 leader_uuid: "0c55efb174ac4320b6a7794e95ec14fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c55efb174ac4320b6a7794e95ec14fb" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40245 } } }
I20260807 02:21:43.363667 5078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb [sys.catalog]: This master's current role is: LEADER
I20260807 02:21:43.368970 5090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:21:43.377744 5090 catalog_manager.cc:1347] Generated new cluster ID: d3c83050c6fd4365a6b347f9817f1e0e
I20260807 02:21:43.377861 5090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:21:43.402904 5090 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:21:43.404451 5090 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:21:43.411927 5090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c55efb174ac4320b6a7794e95ec14fb: Generated new TSK 0
I20260807 02:21:43.413029 5090 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:21:43.455675 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40245
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:43.622288 5097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:43.622695 5097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:43.622783 5097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:43.627408 5097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:43.627533 5097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:43.627641 5097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:21:43.633597 5097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40245
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:43.634969 5097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:43.636094 5097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:43.649366 5103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:43.649605 5105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:43.649317 5102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:43.649861 5097 server_base.cc:1034] running on GCE node
I20260807 02:21:43.650318 5097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:43.651047 5097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:43.652319 5097 hybrid_clock.cc:648] HybridClock initialized: now 1786069303652260 us; error 39 us; skew 500 ppm
Aug 07 02:21:43 dist-test-slave-69rg krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:43.656301 5097 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:21:43.657322 5097 webserver.cc:460] Webserver started at http://127.4.221.65:40345/ using document root <none> and password file <none>
I20260807 02:21:43.657981 5097 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:43.658078 5097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:43.658334 5097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:43.660967 5097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "8f790d7a8c034b599de25f56bad4afd6"
format_stamp: "Formatted at 2026-08-07 02:21:43 on dist-test-slave-69rg"
server_key: "93613c52b3875b4e2cfed638e36cc665"
server_key_iv: "3559a61a1051b194f63d23ff76365ae4"
server_key_version: "encryptionkey@0"
I20260807 02:21:43.661741 5097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "8f790d7a8c034b599de25f56bad4afd6"
format_stamp: "Formatted at 2026-08-07 02:21:43 on dist-test-slave-69rg"
server_key: "93613c52b3875b4e2cfed638e36cc665"
server_key_iv: "3559a61a1051b194f63d23ff76365ae4"
server_key_version: "encryptionkey@0"
I20260807 02:21:43.666402 5097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 02:21:43.669170 5112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:43.670547 5097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:21:43.670697 5097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "8f790d7a8c034b599de25f56bad4afd6"
format_stamp: "Formatted at 2026-08-07 02:21:43 on dist-test-slave-69rg"
server_key: "93613c52b3875b4e2cfed638e36cc665"
server_key_iv: "3559a61a1051b194f63d23ff76365ae4"
server_key_version: "encryptionkey@0"
I20260807 02:21:43.670830 5097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:43.688468 5097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:43.692436 5097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:43.692703 5097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:43.693555 5097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:21:43.694765 5097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:21:43.694844 5097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:43.694912 5097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:21:43.694943 5097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:43.712442 5097 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:39509
I20260807 02:21:43.712544 5229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:39509 every 8 connection(s)
I20260807 02:21:43.713587 5097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:21:43.715508 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 5097
I20260807 02:21:43.715622 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:21:43.715957 4981 external_mini_cluster.cc:1428] Setting key b94b167899ad716406d4fc12c946ec4f
Aug 07 02:21:43 dist-test-slave-69rg krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
I20260807 02:21:43.733108 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:43.716190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:37075 (local address 127.4.221.126:40245)
0807 02:21:43.716350 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:43.716354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:43.717650 (+ 1296us) server_negotiation.cc:407] Connection header received
0807 02:21:43.719755 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:43.719761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:43.719834 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:43.719936 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:43.722295 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:43.722860 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:43.724324 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:43.724588 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:43.724724 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:43.728441 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:43.728448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:43.728453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:43.728511 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:43.730660 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:43.731313 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:43.731317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:43.731319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:43.731421 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:43.731883 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:43.731891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:43.731895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:43.732138 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:43.732224 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:43.732851 (+ 627us) server_negotiation.cc:299] Negotiation successful
0807 02:21:43.732974 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 02:21:43.734282 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:43.716691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40245 (local address 127.4.221.65:37075)
0807 02:21:43.717470 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:43.717530 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:43.719472 (+ 1942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:43.720306 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:43.720318 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:43.721115 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:43.722079 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:43.722098 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:43.723036 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:43.723040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:43.724164 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:43.724175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:43.724788 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:43.724793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:43.725119 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:43.725970 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:43.725998 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:43.728177 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:43.730827 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:43.730834 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:43.730851 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:43.731171 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:43.731523 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:43.731525 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:43.731527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:43.731745 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:43.732237 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:43.732242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:43.732547 (+ 305us) client_negotiation.cc:769] Sending connection context
0807 02:21:43.732753 (+ 206us) client_negotiation.cc:240] Negotiation successful
0807 02:21:43.733134 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":218,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
I20260807 02:21:43.735963 5230 heartbeater.cc:347] Connected to a master server at 127.4.221.126:40245
I20260807 02:21:43.736244 5230 heartbeater.cc:464] Registering TS with master...
I20260807 02:21:43.736819 5230 heartbeater.cc:511] Master 127.4.221.126:40245 requested a full tablet report, sending...
I20260807 02:21:43.739708 5035 ts_manager.cc:194] Registered new tserver with Master: 8f790d7a8c034b599de25f56bad4afd6 (127.4.221.65:39509)
I20260807 02:21:43.742084 5035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:37075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:21:43.757428 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40245
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:43.912350 5235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:43.912657 5235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:43.912707 5235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:43.917186 5235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:43.917260 5235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:43.917327 5235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:21:43.922930 5235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40245
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:43.924044 5235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:43.925076 5235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:43.936213 5240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:43.936257 5241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:43.936215 5243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:43.936347 5235 server_base.cc:1034] running on GCE node
I20260807 02:21:43.936985 5235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:43.937706 5235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:43.938903 5235 hybrid_clock.cc:648] HybridClock initialized: now 1786069303938869 us; error 50 us; skew 500 ppm
Aug 07 02:21:43 dist-test-slave-69rg krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:43.942057 5235 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:21:43.943040 5235 webserver.cc:460] Webserver started at http://127.4.221.66:42683/ using document root <none> and password file <none>
I20260807 02:21:43.943667 5235 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:43.943770 5235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:43.944020 5235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:43.946357 5235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "1913dcb298a749dcbab5fcb8536f8d30"
format_stamp: "Formatted at 2026-08-07 02:21:43 on dist-test-slave-69rg"
server_key: "bba6e2cbc2be2b5df0d349e6fcb4af79"
server_key_iv: "b026c478203e499902880c0fd8213caa"
server_key_version: "encryptionkey@0"
I20260807 02:21:43.946964 5235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "1913dcb298a749dcbab5fcb8536f8d30"
format_stamp: "Formatted at 2026-08-07 02:21:43 on dist-test-slave-69rg"
server_key: "bba6e2cbc2be2b5df0d349e6fcb4af79"
server_key_iv: "b026c478203e499902880c0fd8213caa"
server_key_version: "encryptionkey@0"
I20260807 02:21:43.951149 5235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:21:43.953910 5250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:43.955267 5235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 02:21:43.955422 5235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "1913dcb298a749dcbab5fcb8536f8d30"
format_stamp: "Formatted at 2026-08-07 02:21:43 on dist-test-slave-69rg"
server_key: "bba6e2cbc2be2b5df0d349e6fcb4af79"
server_key_iv: "b026c478203e499902880c0fd8213caa"
server_key_version: "encryptionkey@0"
I20260807 02:21:43.955555 5235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:43.989755 5235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:43.993463 5235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:43.993698 5235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:43.994401 5235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:21:43.995296 5235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:21:43.995366 5235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:43.995441 5235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:21:43.995484 5235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:44.006330 5235 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:46507
I20260807 02:21:44.006374 5366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:46507 every 8 connection(s)
I20260807 02:21:44.007402 5235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:21:44.017742 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 5235
I20260807 02:21:44.017926 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:21:44.018229 4981 external_mini_cluster.cc:1428] Setting key 918cc8e1e8940177daf963ccd69e8553
Aug 07 02:21:44 dist-test-slave-69rg krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069303, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:44.023310 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:44.009525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:33609 (local address 127.4.221.126:40245)
0807 02:21:44.009681 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:44.009684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:44.011255 (+ 1571us) server_negotiation.cc:407] Connection header received
0807 02:21:44.012254 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:44.012259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:44.012335 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:44.012447 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:44.013750 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:44.014490 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:44.015698 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:44.015943 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:44.016064 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:44.019053 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:44.019057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:44.019060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:44.019094 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:44.020820 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:44.021400 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:44.021404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:44.021406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:44.021494 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:44.021924 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:44.021927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:44.021929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:44.022236 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:44.022360 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:44.022935 (+ 575us) server_negotiation.cc:299] Negotiation successful
0807 02:21:44.023096 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 02:21:44.024343 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:44.009925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40245 (local address 127.4.221.66:33609)
0807 02:21:44.011069 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:44.011119 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:44.012011 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:44.012645 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:44.012652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:44.013067 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:44.013582 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:44.013597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:44.014634 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:44.014641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:44.015559 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:44.015568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:44.016118 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:44.016122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:44.016314 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:44.016984 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:44.017002 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:44.018873 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:44.020956 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:44.020963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:44.020982 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:44.021284 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:44.021594 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:44.021598 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:44.021600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:44.021777 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:44.022364 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:44.022371 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:44.022659 (+ 288us) client_negotiation.cc:769] Sending connection context
0807 02:21:44.022833 (+ 174us) client_negotiation.cc:240] Negotiation successful
0807 02:21:44.023177 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"thread_start_us":744,"threads_started":1}
I20260807 02:21:44.025671 5367 heartbeater.cc:347] Connected to a master server at 127.4.221.126:40245
I20260807 02:21:44.025997 5367 heartbeater.cc:464] Registering TS with master...
I20260807 02:21:44.026553 5367 heartbeater.cc:511] Master 127.4.221.126:40245 requested a full tablet report, sending...
I20260807 02:21:44.027873 5035 ts_manager.cc:194] Registered new tserver with Master: 1913dcb298a749dcbab5fcb8536f8d30 (127.4.221.66:46507)
I20260807 02:21:44.028535 5035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:33609
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:21:44.053643 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40245
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:41183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:44.177192 5372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:44.177469 5372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:44.177515 5372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:44.181667 5372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:44.181734 5372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:44.181797 5372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:21:44.187515 5372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:41183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40245
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:44.188681 5372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:44.189726 5372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:44.200529 5377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:44.200645 5372 server_base.cc:1034] running on GCE node
W20260807 02:21:44.200519 5380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:44.200875 5378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:44.201314 5372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:44.202144 5372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:44.203325 5372 hybrid_clock.cc:648] HybridClock initialized: now 1786069304203310 us; error 30 us; skew 500 ppm
Aug 07 02:21:44 dist-test-slave-69rg krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:44.206409 5372 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:21:44.207358 5372 webserver.cc:460] Webserver started at http://127.4.221.67:43463/ using document root <none> and password file <none>
I20260807 02:21:44.207913 5372 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:44.207975 5372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:44.208158 5372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:44.210413 5372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "e201439bf854415c897d91cc03a1157e"
format_stamp: "Formatted at 2026-08-07 02:21:44 on dist-test-slave-69rg"
server_key: "0723ad14cad2168eef7acf03fa1dd158"
server_key_iv: "3022a0d3eaa9b4417241dfce5b34966a"
server_key_version: "encryptionkey@0"
I20260807 02:21:44.211004 5372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "e201439bf854415c897d91cc03a1157e"
format_stamp: "Formatted at 2026-08-07 02:21:44 on dist-test-slave-69rg"
server_key: "0723ad14cad2168eef7acf03fa1dd158"
server_key_iv: "3022a0d3eaa9b4417241dfce5b34966a"
server_key_version: "encryptionkey@0"
I20260807 02:21:44.215250 5372 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260807 02:21:44.218030 5387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:44.219383 5372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 02:21:44.219543 5372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "e201439bf854415c897d91cc03a1157e"
format_stamp: "Formatted at 2026-08-07 02:21:44 on dist-test-slave-69rg"
server_key: "0723ad14cad2168eef7acf03fa1dd158"
server_key_iv: "3022a0d3eaa9b4417241dfce5b34966a"
server_key_version: "encryptionkey@0"
I20260807 02:21:44.219681 5372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:44.247040 5372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:44.250825 5372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:44.251060 5372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:44.251735 5372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:21:44.252658 5372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:21:44.252733 5372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:44.252808 5372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:21:44.252862 5372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:44.262986 5372 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:36077
I20260807 02:21:44.263053 5503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:36077 every 8 connection(s)
I20260807 02:21:44.264020 5372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:21:44.274114 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 5372
I20260807 02:21:44.274228 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
Aug 07 02:21:44 dist-test-slave-69rg krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069304, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:44.274535 4981 external_mini_cluster.cc:1428] Setting key 2d09873ee0f83ca4c550e529d037fb72
I20260807 02:21:44.279578 5073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:44.266127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:34571 (local address 127.4.221.126:40245)
0807 02:21:44.266287 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:44.266292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:44.267267 (+ 975us) server_negotiation.cc:407] Connection header received
0807 02:21:44.268715 (+ 1448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:44.268719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:44.268781 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:44.268889 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:44.270217 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:44.270767 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:44.271966 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:44.272191 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:44.272301 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:44.275390 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:44.275396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:44.275399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:44.275432 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:44.277209 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:44.277881 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:44.277885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:44.277887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:44.277996 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:44.278459 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:44.278462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:44.278463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:44.278688 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:44.278770 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:44.279326 (+ 556us) server_negotiation.cc:299] Negotiation successful
0807 02:21:44.279443 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 02:21:44.280455 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:44.266572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40245 (local address 127.4.221.67:34571)
0807 02:21:44.267095 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:44.267140 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:44.268480 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:44.269082 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:44.269088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:44.269482 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:44.270059 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:44.270075 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:44.270910 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:44.270914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:44.271817 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:44.271828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:44.272350 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:44.272354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:44.272527 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:44.273287 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:44.273304 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:44.275148 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:44.277354 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:44.277362 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:44.277387 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:44.277678 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:44.278123 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:44.278126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:44.278129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:44.278348 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:44.278782 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:44.278788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:44.279058 (+ 270us) client_negotiation.cc:769] Sending connection context
0807 02:21:44.279255 (+ 197us) client_negotiation.cc:240] Negotiation successful
0807 02:21:44.279545 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":152,"threads_started":1}
I20260807 02:21:44.281683 5504 heartbeater.cc:347] Connected to a master server at 127.4.221.126:40245
I20260807 02:21:44.282008 5504 heartbeater.cc:464] Registering TS with master...
I20260807 02:21:44.282598 5504 heartbeater.cc:511] Master 127.4.221.126:40245 requested a full tablet report, sending...
I20260807 02:21:44.283720 5035 ts_manager.cc:194] Registered new tserver with Master: e201439bf854415c897d91cc03a1157e (127.4.221.67:36077)
I20260807 02:21:44.284332 5035 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:34571
I20260807 02:21:44.290488 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 02:21:44 dist-test-slave-69rg krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069304, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 02:21:44 dist-test-slave-69rg krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069304, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.221.65@KRBTEST.COM
I20260807 02:21:44.321347 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:44.310737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.4.221.65:39509)
0807 02:21:44.311090 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:44.311100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:44.311124 (+ 24us) server_negotiation.cc:407] Connection header received
0807 02:21:44.311177 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:44.311182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:44.311428 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:44.311560 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:44.312328 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:44.313082 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:44.314023 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:44.314408 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:44.314495 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:44.317227 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:44.317241 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:44.317256 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:44.317286 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:44.319511 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:44.319972 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:44.319979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:44.319981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:44.320081 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:44.320342 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:44.320345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:44.320347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:44.320648 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:44.320864 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:44.321002 (+ 138us) server_negotiation.cc:299] Negotiation successful
0807 02:21:44.321163 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":139,"threads_started":1}
W20260807 02:21:44.322559 5144 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49114
Aug 07 02:21:44 dist-test-slave-69rg krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 02:21:44 dist-test-slave-69rg krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.65@KRBTEST.COM
I20260807 02:21:44.353924 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:44.345220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.4.221.65:39509)
0807 02:21:44.345401 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:44.345408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:44.345533 (+ 125us) server_negotiation.cc:407] Connection header received
0807 02:21:44.345723 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:44.345727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:44.345781 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:44.345922 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:44.346678 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:44.347203 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:44.348051 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:44.348307 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:44.348420 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:44.350948 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:44.350952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:44.350954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:44.350979 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:44.352331 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:44.352754 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:44.352757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:44.352758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:44.352842 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:44.353082 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:44.353084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:44.353085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:44.353288 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:44.353437 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:44.353617 (+ 180us) server_negotiation.cc:299] Negotiation successful
0807 02:21:44.353726 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 02:21:44.355185 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 5097
I20260807 02:21:44.362509 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 5235
I20260807 02:21:44.369953 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 5372
I20260807 02:21:44.376482 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 5003
2026-08-07T02:21:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2529 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:21:44 dist-test-slave-69rg krb5kdc[5518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:21:44 dist-test-slave-69rg krb5kdc[5518](info): set up 2 sockets
Aug 07 02:21:44 dist-test-slave-69rg krb5kdc[5518](info): commencing operation
krb5kdc: starting...
W20260807 02:21:45.445477 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:21:45 dist-test-slave-69rg krb5kdc[5518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:21:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:21:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:21:45.589403 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:38471
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:38471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:45.709695 5531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:45.710103 5531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:45.710193 5531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:45.714564 5531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:21:45.714676 5531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:45.714715 5531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:45.714756 5531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:21:45.714788 5531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:21:45.720728 5531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:38471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:38471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:45.722030 5531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:45.723105 5531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:45.732131 5537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:45.732182 5536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:45.732153 5539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:45.732460 5531 server_base.cc:1034] running on GCE node
I20260807 02:21:45.733157 5531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:45.734427 5531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:45.735601 5531 hybrid_clock.cc:648] HybridClock initialized: now 1786069305735574 us; error 51 us; skew 500 ppm
Aug 07 02:21:45 dist-test-slave-69rg krb5kdc[5518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069305, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:45.743496 5531 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:21:45.744520 5531 webserver.cc:460] Webserver started at http://127.4.221.126:42525/ using document root <none> and password file <none>
I20260807 02:21:45.745102 5531 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:45.745168 5531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:45.745357 5531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:45.747725 5531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "092f2639d605486fbbf633654ca44c11"
format_stamp: "Formatted at 2026-08-07 02:21:45 on dist-test-slave-69rg"
server_key: "5f8194c90846697af1262e355459ba33"
server_key_iv: "bf88a6b878030521105ded271ea01896"
server_key_version: "encryptionkey@0"
I20260807 02:21:45.748293 5531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "092f2639d605486fbbf633654ca44c11"
format_stamp: "Formatted at 2026-08-07 02:21:45 on dist-test-slave-69rg"
server_key: "5f8194c90846697af1262e355459ba33"
server_key_iv: "bf88a6b878030521105ded271ea01896"
server_key_version: "encryptionkey@0"
I20260807 02:21:45.752486 5531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:21:45.755184 5546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:45.756479 5531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 02:21:45.756590 5531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "092f2639d605486fbbf633654ca44c11"
format_stamp: "Formatted at 2026-08-07 02:21:45 on dist-test-slave-69rg"
server_key: "5f8194c90846697af1262e355459ba33"
server_key_iv: "bf88a6b878030521105ded271ea01896"
server_key_version: "encryptionkey@0"
I20260807 02:21:45.756711 5531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:45.783739 5531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:45.787388 5531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:45.787565 5531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:45.795239 5531 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:38471
I20260807 02:21:45.795269 5599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:38471 every 8 connection(s)
I20260807 02:21:45.796347 5531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:21:45.797549 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 5531
I20260807 02:21:45.797640 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:21:45.798000 4981 external_mini_cluster.cc:1428] Setting key 75abbee3226c4350db0c041f7e739019
I20260807 02:21:45.800019 5600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:45.807740 5600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11: Bootstrap starting.
Aug 07 02:21:45 dist-test-slave-69rg krb5kdc[5518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:45.811250 5600 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:45.812242 5600 log.cc:824] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11: Log is configured to *not* fsync() on all Append() calls
I20260807 02:21:45.814822 5600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11: No bootstrap required, opened a new log
I20260807 02:21:45.815119 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:45.799620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.4.221.126:38471)
0807 02:21:45.800199 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:45.800221 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:45.800269 (+ 48us) server_negotiation.cc:407] Connection header received
0807 02:21:45.801185 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:45.801211 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:45.801673 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:45.802115 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:45.802890 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:45.804441 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:45.805313 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:45.806122 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:45.806310 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:45.809311 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:45.809322 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:45.809347 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:45.809394 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:45.811716 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:45.812210 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:45.812216 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:45.812223 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:45.812412 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:45.812722 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:45.812725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:45.812727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:45.813220 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:45.813427 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:45.813772 (+ 345us) server_negotiation.cc:299] Negotiation successful
0807 02:21:45.814114 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":171,"threads_started":1}
I20260807 02:21:45.819363 5600 raft_consensus.cc:348] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f2639d605486fbbf633654ca44c11" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38471 } }
I20260807 02:21:45.819609 5600 raft_consensus.cc:374] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:45.819646 5600 raft_consensus.cc:729] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092f2639d605486fbbf633654ca44c11, State: Initialized, Role: FOLLOWER
I20260807 02:21:45.820013 5600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f2639d605486fbbf633654ca44c11" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38471 } }
I20260807 02:21:45.820127 5600 raft_consensus.cc:388] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:21:45.820212 5600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:21:45.820312 5600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:45.821346 5600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f2639d605486fbbf633654ca44c11" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38471 } }
I20260807 02:21:45.821732 5600 leader_election.cc:302] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 092f2639d605486fbbf633654ca44c11; no voters:
I20260807 02:21:45.822139 5600 leader_election.cc:288] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:21:45.822252 5605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:21:45.822541 5605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [term 1 LEADER]: Becoming Leader. State: Replica: 092f2639d605486fbbf633654ca44c11, State: Running, Role: LEADER
I20260807 02:21:45.822872 5605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f2639d605486fbbf633654ca44c11" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38471 } }
I20260807 02:21:45.823482 5600 sys_catalog.cc:564] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:21:45.824610 5607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 092f2639d605486fbbf633654ca44c11. Latest consensus state: current_term: 1 leader_uuid: "092f2639d605486fbbf633654ca44c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f2639d605486fbbf633654ca44c11" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38471 } } }
I20260807 02:21:45.824757 5607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [sys.catalog]: This master's current role is: LEADER
I20260807 02:21:45.825039 5606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "092f2639d605486fbbf633654ca44c11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "092f2639d605486fbbf633654ca44c11" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38471 } } }
I20260807 02:21:45.825145 5606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11 [sys.catalog]: This master's current role is: LEADER
I20260807 02:21:45.825284 5615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:21:45.828591 5615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:21:45.835454 5615 catalog_manager.cc:1347] Generated new cluster ID: 9dd7438624044b8c89a346959c1dbefb
I20260807 02:21:45.835584 5615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:21:45.856829 5615 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:21:45.858345 5615 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:21:45.865661 5615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 092f2639d605486fbbf633654ca44c11: Generated new TSK 0
I20260807 02:21:45.866753 5615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:21:45.907425 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:38471
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:46.065021 5627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:46.065482 5627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:46.065553 5627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:46.071527 5627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:46.071611 5627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:46.071741 5627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:21:46.078246 5627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:38471
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:46.079433 5627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:46.080493 5627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:46.091159 5633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:46.091159 5635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:46.091161 5632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:46.091445 5627 server_base.cc:1034] running on GCE node
I20260807 02:21:46.091908 5627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:46.092624 5627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:46.093801 5627 hybrid_clock.cc:648] HybridClock initialized: now 1786069306093788 us; error 26 us; skew 500 ppm
Aug 07 02:21:46 dist-test-slave-69rg krb5kdc[5518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069306, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:46.096899 5627 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:21:46.097898 5627 webserver.cc:460] Webserver started at http://127.4.221.65:46117/ using document root <none> and password file <none>
I20260807 02:21:46.098495 5627 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:46.098558 5627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:46.098857 5627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:46.101037 5627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "f380b9f34b2e4beeadb0347c00ff3b40"
format_stamp: "Formatted at 2026-08-07 02:21:46 on dist-test-slave-69rg"
server_key: "e5fff5d9e8609775b868957292488a45"
server_key_iv: "8145facf45b44a044fcfd1357a21c7e4"
server_key_version: "encryptionkey@0"
I20260807 02:21:46.101593 5627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "f380b9f34b2e4beeadb0347c00ff3b40"
format_stamp: "Formatted at 2026-08-07 02:21:46 on dist-test-slave-69rg"
server_key: "e5fff5d9e8609775b868957292488a45"
server_key_iv: "8145facf45b44a044fcfd1357a21c7e4"
server_key_version: "encryptionkey@0"
I20260807 02:21:46.105608 5627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260807 02:21:46.108376 5642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:46.109728 5627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:21:46.109906 5627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "f380b9f34b2e4beeadb0347c00ff3b40"
format_stamp: "Formatted at 2026-08-07 02:21:46 on dist-test-slave-69rg"
server_key: "e5fff5d9e8609775b868957292488a45"
server_key_iv: "8145facf45b44a044fcfd1357a21c7e4"
server_key_version: "encryptionkey@0"
I20260807 02:21:46.110040 5627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:46.133301 5627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:46.137074 5627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:46.137315 5627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:46.138013 5627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:21:46.138955 5627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:21:46.139005 5627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:46.139081 5627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:21:46.139111 5627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:46.148646 5627 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:35543
I20260807 02:21:46.148659 5757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:35543 every 8 connection(s)
I20260807 02:21:46.149689 5627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:21:46.151518 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 5627
I20260807 02:21:46.151644 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:21:46.151947 4981 external_mini_cluster.cc:1428] Setting key cfd5dff3c24abd5f9242bf58b862a06f
Aug 07 02:21:46 dist-test-slave-69rg krb5kdc[5518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069306, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
I20260807 02:21:46.165377 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.151731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:38519 (local address 127.4.221.126:38471)
0807 02:21:46.151930 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:46.151936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:46.152950 (+ 1014us) server_negotiation.cc:407] Connection header received
0807 02:21:46.154469 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:46.154473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:46.154532 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:46.154621 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:46.155896 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.156453 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.157653 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.157976 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.158096 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.161230 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:46.161234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:46.161236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:46.161264 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:46.162984 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:46.163687 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:46.163690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:46.163692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:46.163779 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:46.164092 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:46.164094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:46.164095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:46.164305 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:46.164395 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:46.164994 (+ 599us) server_negotiation.cc:299] Negotiation successful
0807 02:21:46.165182 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260807 02:21:46.166508 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.152234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:38471 (local address 127.4.221.65:38519)
0807 02:21:46.152769 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:46.152837 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:46.154236 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:46.154823 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:46.154831 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:46.155215 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:46.155732 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.155747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.156602 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.156606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.157518 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.157527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.158161 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.158166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.158351 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.159048 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:46.159065 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:46.161001 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:46.163109 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:46.163114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:46.163129 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:46.163566 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:46.163867 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:46.163869 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:46.163870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:46.164010 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:46.164438 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:46.164443 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:46.164742 (+ 299us) client_negotiation.cc:769] Sending connection context
0807 02:21:46.164912 (+ 170us) client_negotiation.cc:240] Negotiation successful
0807 02:21:46.165358 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":156,"threads_started":1}
I20260807 02:21:46.168081 5758 heartbeater.cc:347] Connected to a master server at 127.4.221.126:38471
I20260807 02:21:46.168398 5758 heartbeater.cc:464] Registering TS with master...
I20260807 02:21:46.168967 5758 heartbeater.cc:511] Master 127.4.221.126:38471 requested a full tablet report, sending...
I20260807 02:21:46.170756 5563 ts_manager.cc:194] Registered new tserver with Master: f380b9f34b2e4beeadb0347c00ff3b40 (127.4.221.65:35543)
I20260807 02:21:46.171968 5563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:38519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:21:46.185182 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:38471
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:46.307853 5763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:46.308172 5763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:46.308223 5763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:46.312476 5763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:46.312559 5763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:46.312626 5763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:21:46.318678 5763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:38471
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:46.319989 5763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:46.321044 5763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:46.332082 5768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:46.332016 5771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:46.332024 5769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:46.332407 5763 server_base.cc:1034] running on GCE node
I20260807 02:21:46.332851 5763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:46.333554 5763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:46.334740 5763 hybrid_clock.cc:648] HybridClock initialized: now 1786069306334698 us; error 53 us; skew 500 ppm
Aug 07 02:21:46 dist-test-slave-69rg krb5kdc[5518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069306, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:46.337690 5763 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:21:46.338724 5763 webserver.cc:460] Webserver started at http://127.4.221.66:32895/ using document root <none> and password file <none>
I20260807 02:21:46.339334 5763 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:46.339433 5763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:46.339680 5763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:46.341960 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "87b20570250544958669924a06841bc7"
format_stamp: "Formatted at 2026-08-07 02:21:46 on dist-test-slave-69rg"
server_key: "6ed8cfd99b7896f114a10e597b71f0bf"
server_key_iv: "6e9c3d4ce0441254423039094a3ee9c4"
server_key_version: "encryptionkey@0"
I20260807 02:21:46.342582 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "87b20570250544958669924a06841bc7"
format_stamp: "Formatted at 2026-08-07 02:21:46 on dist-test-slave-69rg"
server_key: "6ed8cfd99b7896f114a10e597b71f0bf"
server_key_iv: "6e9c3d4ce0441254423039094a3ee9c4"
server_key_version: "encryptionkey@0"
I20260807 02:21:46.346791 5763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 02:21:46.349390 5778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:46.350726 5763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:21:46.350973 5763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "87b20570250544958669924a06841bc7"
format_stamp: "Formatted at 2026-08-07 02:21:46 on dist-test-slave-69rg"
server_key: "6ed8cfd99b7896f114a10e597b71f0bf"
server_key_iv: "6e9c3d4ce0441254423039094a3ee9c4"
server_key_version: "encryptionkey@0"
I20260807 02:21:46.351105 5763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:46.370810 5763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:46.374037 5763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:46.374253 5763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:46.374861 5763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:21:46.375715 5763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:21:46.375787 5763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:46.375859 5763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:21:46.375911 5763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:46.384749 5763 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:38881
I20260807 02:21:46.384776 5894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:38881 every 8 connection(s)
I20260807 02:21:46.385735 5763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:21:46.394099 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 5763
I20260807 02:21:46.394237 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:21:46.394546 4981 external_mini_cluster.cc:1428] Setting key 44f2e5f3b152bcdb3e8b2473515bda95
Aug 07 02:21:46 dist-test-slave-69rg krb5kdc[5518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069306, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:46.400815 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.387656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:53731 (local address 127.4.221.126:38471)
0807 02:21:46.387807 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:46.387811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:46.388727 (+ 916us) server_negotiation.cc:407] Connection header received
0807 02:21:46.390141 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:46.390145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:46.390196 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:46.390273 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:46.391484 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.392010 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.393121 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.393350 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.393428 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.396510 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:46.396513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:46.396515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:46.396541 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:46.398293 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:46.398841 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:46.398847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:46.398850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:46.398955 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:46.399303 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:46.399308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:46.399310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:46.399590 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:46.399711 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:46.400460 (+ 749us) server_negotiation.cc:299] Negotiation successful
0807 02:21:46.400640 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 02:21:46.402071 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.388048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:38471 (local address 127.4.221.66:53731)
0807 02:21:46.388547 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:46.388592 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:46.389931 (+ 1339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:46.390448 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:46.390456 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:46.390850 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:46.391339 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.391353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.392123 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.392125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.393013 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.393022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.393486 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.393489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.393623 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.394409 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:46.394435 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:46.396322 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:46.398425 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:46.398431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:46.398445 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:46.398734 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:46.399054 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:46.399056 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:46.399057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:46.399206 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:46.399732 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:46.399742 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:46.400132 (+ 390us) client_negotiation.cc:769] Sending connection context
0807 02:21:46.400370 (+ 238us) client_negotiation.cc:240] Negotiation successful
0807 02:21:46.400779 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":122,"threads_started":1}
I20260807 02:21:46.403419 5895 heartbeater.cc:347] Connected to a master server at 127.4.221.126:38471
I20260807 02:21:46.403690 5895 heartbeater.cc:464] Registering TS with master...
I20260807 02:21:46.404232 5895 heartbeater.cc:511] Master 127.4.221.126:38471 requested a full tablet report, sending...
I20260807 02:21:46.405408 5563 ts_manager.cc:194] Registered new tserver with Master: 87b20570250544958669924a06841bc7 (127.4.221.66:38881)
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
I20260807 02:21:46.406081 5563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:53731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:21:46.425707 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:38471
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:46.542425 5900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:46.542737 5900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:46.542785 5900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:46.547040 5900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:46.547119 5900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:46.547185 5900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:21:46.553012 5900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:38471
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:46.554329 5900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:46.555493 5900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:46.566110 5905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:46.566164 5908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:46.566283 5900 server_base.cc:1034] running on GCE node
W20260807 02:21:46.566164 5906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:46.566834 5900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:46.567613 5900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:46.568818 5900 hybrid_clock.cc:648] HybridClock initialized: now 1786069306568784 us; error 47 us; skew 500 ppm
Aug 07 02:21:46 dist-test-slave-69rg krb5kdc[5518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069306, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:46.571942 5900 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:21:46.573010 5900 webserver.cc:460] Webserver started at http://127.4.221.67:40377/ using document root <none> and password file <none>
I20260807 02:21:46.573644 5900 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:46.573745 5900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:46.574028 5900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:46.576377 5900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "a43c60b76ba34fa097ab94b9436e3a7e"
format_stamp: "Formatted at 2026-08-07 02:21:46 on dist-test-slave-69rg"
server_key: "78a53257798ed1633043caea1add64dc"
server_key_iv: "651dad1cdc161ce58758d232738779d2"
server_key_version: "encryptionkey@0"
I20260807 02:21:46.577039 5900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "a43c60b76ba34fa097ab94b9436e3a7e"
format_stamp: "Formatted at 2026-08-07 02:21:46 on dist-test-slave-69rg"
server_key: "78a53257798ed1633043caea1add64dc"
server_key_iv: "651dad1cdc161ce58758d232738779d2"
server_key_version: "encryptionkey@0"
I20260807 02:21:46.581422 5900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 02:21:46.584406 5915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:46.585791 5900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:21:46.586040 5900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "a43c60b76ba34fa097ab94b9436e3a7e"
format_stamp: "Formatted at 2026-08-07 02:21:46 on dist-test-slave-69rg"
server_key: "78a53257798ed1633043caea1add64dc"
server_key_iv: "651dad1cdc161ce58758d232738779d2"
server_key_version: "encryptionkey@0"
I20260807 02:21:46.586192 5900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:46.620371 5900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:46.624516 5900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:46.624785 5900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:46.625499 5900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:21:46.626513 5900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:21:46.626595 5900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:46.626660 5900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:21:46.626714 5900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:46.636125 5900 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:45849
I20260807 02:21:46.636145 6033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:45849 every 8 connection(s)
I20260807 02:21:46.637081 5900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:21:46.644052 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 5900
I20260807 02:21:46.644191 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:21:46.644484 4981 external_mini_cluster.cc:1428] Setting key 528f187d53a4fb491a69e0c030f74ef6
Aug 07 02:21:46 dist-test-slave-69rg krb5kdc[5518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069306, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:46.651677 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.638993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:42195 (local address 127.4.221.126:38471)
0807 02:21:46.639135 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:46.639139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:46.639940 (+ 801us) server_negotiation.cc:407] Connection header received
0807 02:21:46.641317 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:46.641321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:46.641383 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:46.641463 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:46.642704 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.643265 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.644778 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.645144 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.645240 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.648263 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:46.648266 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:46.648268 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:46.648296 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:46.649588 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:46.650159 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:46.650162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:46.650163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:46.650262 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:46.650576 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:46.650578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:46.650579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:46.650773 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:46.650865 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:46.651430 (+ 565us) server_negotiation.cc:299] Negotiation successful
0807 02:21:46.651545 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 02:21:46.652576 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.639320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:38471 (local address 127.4.221.67:42195)
0807 02:21:46.639777 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:46.639822 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:46.641091 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:46.641635 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:46.641641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:46.642052 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:46.642542 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.642556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.643407 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.643417 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.644642 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.644654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.645450 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.645454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.645658 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.646395 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:46.646416 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:46.648086 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:46.649699 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:46.649705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:46.649719 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:46.650071 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:46.650346 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:46.650348 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:46.650349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:46.650487 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:46.650895 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:46.650903 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:46.651193 (+ 290us) client_negotiation.cc:769] Sending connection context
0807 02:21:46.651366 (+ 173us) client_negotiation.cc:240] Negotiation successful
0807 02:21:46.651642 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20260807 02:21:46.653887 6034 heartbeater.cc:347] Connected to a master server at 127.4.221.126:38471
I20260807 02:21:46.654157 6034 heartbeater.cc:464] Registering TS with master...
I20260807 02:21:46.654796 6034 heartbeater.cc:511] Master 127.4.221.126:38471 requested a full tablet report, sending...
I20260807 02:21:46.656210 5563 ts_manager.cc:194] Registered new tserver with Master: a43c60b76ba34fa097ab94b9436e3a7e (127.4.221.67:45849)
I20260807 02:21:46.656872 5563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:42195
I20260807 02:21:46.659950 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:21:46.672371 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.664458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.4.221.126:38471)
0807 02:21:46.664646 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:46.664652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:46.664813 (+ 161us) server_negotiation.cc:407] Connection header received
0807 02:21:46.664973 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:46.664976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:46.665033 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:46.665124 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:46.665918 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.666486 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.667344 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.667584 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.667696 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.669143 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:46.669147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:46.669150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:46.669178 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:46.670541 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:46.670996 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:46.671000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:46.671002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:46.671095 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:46.671387 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:46.671391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:46.671392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:46.671647 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:46.671805 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:46.672054 (+ 249us) server_negotiation.cc:299] Negotiation successful
0807 02:21:46.672190 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 02:21:46.678517 5563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:21:46.682098 5563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:21:46.698849 6047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.692838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42712 (local address 127.4.221.65:35543)
0807 02:21:46.693243 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:46.693250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:46.693278 (+ 28us) server_negotiation.cc:407] Connection header received
0807 02:21:46.693574 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:46.693581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:46.693797 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:46.693989 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:46.694316 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.695518 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.697290 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.698105 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.698240 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.698467 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:46.698584 (+ 117us) server_negotiation.cc:299] Negotiation successful
0807 02:21:46.698679 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":117,"threads_started":1}
I20260807 02:21:46.698836 6045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.692709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:35543 (local address 127.0.0.1:42712)
0807 02:21:46.693158 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:46.693208 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:46.693350 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:46.693978 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:46.693984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:46.694010 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:46.694153 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.694164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.695640 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.695643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.697159 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.697171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.698250 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.698254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.698393 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.698434 (+ 41us) client_negotiation.cc:769] Sending connection context
0807 02:21:46.698554 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 02:21:46.698679 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":83,"thread_start_us":97,"threads_started":1}
I20260807 02:21:46.699232 6046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.692790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:45849 (local address 127.0.0.1:45072)
0807 02:21:46.693449 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:46.693463 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:46.693555 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:46.695116 (+ 1561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:46.695119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:46.695139 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:46.695243 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.695248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.696380 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.696384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.697673 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.697684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.698813 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.698816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.698913 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.698937 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 02:21:46.698988 (+ 51us) client_negotiation.cc:240] Negotiation successful
0807 02:21:46.699068 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":42,"thread_start_us":114,"threads_started":1}
I20260807 02:21:46.699548 6049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.694080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.4.221.66:38881)
0807 02:21:46.694457 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:46.694469 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:46.694492 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:21:46.694569 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:46.694574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:46.694725 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:46.694882 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:46.695176 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.696246 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.697680 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.698567 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.698676 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.698844 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:46.699302 (+ 458us) server_negotiation.cc:299] Negotiation successful
0807 02:21:46.699407 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":127,"threads_started":1}
I20260807 02:21:46.699232 6048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.693446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:38881 (local address 127.0.0.1:57526)
0807 02:21:46.693696 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:46.693711 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:46.693809 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:46.694865 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:46.694868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:46.694887 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:46.695007 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.695018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.696400 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.696404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.697562 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.697571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.698898 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.698901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.698992 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.699010 (+ 18us) client_negotiation.cc:769] Sending connection context
0807 02:21:46.699068 (+ 58us) client_negotiation.cc:240] Negotiation successful
0807 02:21:46.699117 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":69,"thread_start_us":110,"threads_started":1}
I20260807 02:21:46.700240 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.694259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.4.221.67:45849)
0807 02:21:46.694643 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:46.694654 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:46.694677 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:21:46.694747 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:46.694752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:46.694922 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:46.695112 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:46.695360 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.696197 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.697805 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.698678 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.699768 (+ 1090us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.699919 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:46.699969 (+ 50us) server_negotiation.cc:299] Negotiation successful
0807 02:21:46.700074 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":153,"threads_started":1}
I20260807 02:21:46.702756 5964 tablet_service.cc:1467] Processing CreateTablet for tablet b247c36a730649268434e0da43c490de (DEFAULT_TABLE table=test-table [id=2d3b49405a4b4c4c8eb941a5a2582cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:46.702770 5691 tablet_service.cc:1467] Processing CreateTablet for tablet b247c36a730649268434e0da43c490de (DEFAULT_TABLE table=test-table [id=2d3b49405a4b4c4c8eb941a5a2582cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:46.704016 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b247c36a730649268434e0da43c490de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:46.704022 5964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b247c36a730649268434e0da43c490de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:46.704493 5827 tablet_service.cc:1467] Processing CreateTablet for tablet b247c36a730649268434e0da43c490de (DEFAULT_TABLE table=test-table [id=2d3b49405a4b4c4c8eb941a5a2582cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:46.705482 5827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b247c36a730649268434e0da43c490de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:46.710638 6051 tablet_bootstrap.cc:492] T b247c36a730649268434e0da43c490de P a43c60b76ba34fa097ab94b9436e3a7e: Bootstrap starting.
I20260807 02:21:46.711992 6052 tablet_bootstrap.cc:492] T b247c36a730649268434e0da43c490de P f380b9f34b2e4beeadb0347c00ff3b40: Bootstrap starting.
I20260807 02:21:46.712397 6053 tablet_bootstrap.cc:492] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7: Bootstrap starting.
I20260807 02:21:46.714020 6052 tablet_bootstrap.cc:654] T b247c36a730649268434e0da43c490de P f380b9f34b2e4beeadb0347c00ff3b40: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:46.714958 6052 log.cc:824] T b247c36a730649268434e0da43c490de P f380b9f34b2e4beeadb0347c00ff3b40: Log is configured to *not* fsync() on all Append() calls
I20260807 02:21:46.715221 6053 tablet_bootstrap.cc:654] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:46.715852 6053 log.cc:824] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7: Log is configured to *not* fsync() on all Append() calls
I20260807 02:21:46.717231 6051 tablet_bootstrap.cc:654] T b247c36a730649268434e0da43c490de P a43c60b76ba34fa097ab94b9436e3a7e: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:46.718046 6053 tablet_bootstrap.cc:492] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7: No bootstrap required, opened a new log
I20260807 02:21:46.718091 6051 log.cc:824] T b247c36a730649268434e0da43c490de P a43c60b76ba34fa097ab94b9436e3a7e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:21:46.718305 6053 ts_tablet_manager.cc:1397] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 02:21:46.719532 6052 tablet_bootstrap.cc:492] T b247c36a730649268434e0da43c490de P f380b9f34b2e4beeadb0347c00ff3b40: No bootstrap required, opened a new log
I20260807 02:21:46.719745 6052 ts_tablet_manager.cc:1397] T b247c36a730649268434e0da43c490de P f380b9f34b2e4beeadb0347c00ff3b40: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260807 02:21:46.720098 6051 tablet_bootstrap.cc:492] T b247c36a730649268434e0da43c490de P a43c60b76ba34fa097ab94b9436e3a7e: No bootstrap required, opened a new log
I20260807 02:21:46.720290 6051 ts_tablet_manager.cc:1397] T b247c36a730649268434e0da43c490de P a43c60b76ba34fa097ab94b9436e3a7e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260807 02:21:46.722419 6053 raft_consensus.cc:348] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f380b9f34b2e4beeadb0347c00ff3b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35543 } } peers { permanent_uuid: "a43c60b76ba34fa097ab94b9436e3a7e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45849 } } peers { permanent_uuid: "87b20570250544958669924a06841bc7" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 38881 } }
I20260807 02:21:46.722724 6053 raft_consensus.cc:374] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:46.722783 6053 raft_consensus.cc:729] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b20570250544958669924a06841bc7, State: Initialized, Role: FOLLOWER
I20260807 02:21:46.723325 6053 consensus_queue.cc:260] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f380b9f34b2e4beeadb0347c00ff3b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35543 } } peers { permanent_uuid: "a43c60b76ba34fa097ab94b9436e3a7e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45849 } } peers { permanent_uuid: "87b20570250544958669924a06841bc7" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 38881 } }
I20260807 02:21:46.723933 6052 raft_consensus.cc:348] T b247c36a730649268434e0da43c490de P f380b9f34b2e4beeadb0347c00ff3b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f380b9f34b2e4beeadb0347c00ff3b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35543 } } peers { permanent_uuid: "a43c60b76ba34fa097ab94b9436e3a7e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45849 } } peers { permanent_uuid: "87b20570250544958669924a06841bc7" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 38881 } }
I20260807 02:21:46.724188 6052 raft_consensus.cc:374] T b247c36a730649268434e0da43c490de P f380b9f34b2e4beeadb0347c00ff3b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:46.724323 6052 raft_consensus.cc:729] T b247c36a730649268434e0da43c490de P f380b9f34b2e4beeadb0347c00ff3b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f380b9f34b2e4beeadb0347c00ff3b40, State: Initialized, Role: FOLLOWER
I20260807 02:21:46.724215 6051 raft_consensus.cc:348] T b247c36a730649268434e0da43c490de P a43c60b76ba34fa097ab94b9436e3a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f380b9f34b2e4beeadb0347c00ff3b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35543 } } peers { permanent_uuid: "a43c60b76ba34fa097ab94b9436e3a7e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45849 } } peers { permanent_uuid: "87b20570250544958669924a06841bc7" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 38881 } }
I20260807 02:21:46.724464 6051 raft_consensus.cc:374] T b247c36a730649268434e0da43c490de P a43c60b76ba34fa097ab94b9436e3a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:46.724515 6051 raft_consensus.cc:729] T b247c36a730649268434e0da43c490de P a43c60b76ba34fa097ab94b9436e3a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a43c60b76ba34fa097ab94b9436e3a7e, State: Initialized, Role: FOLLOWER
I20260807 02:21:46.724630 6053 ts_tablet_manager.cc:1428] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 02:21:46.724957 6052 consensus_queue.cc:260] T b247c36a730649268434e0da43c490de P f380b9f34b2e4beeadb0347c00ff3b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f380b9f34b2e4beeadb0347c00ff3b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35543 } } peers { permanent_uuid: "a43c60b76ba34fa097ab94b9436e3a7e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45849 } } peers { permanent_uuid: "87b20570250544958669924a06841bc7" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 38881 } }
I20260807 02:21:46.725036 6051 consensus_queue.cc:260] T b247c36a730649268434e0da43c490de P a43c60b76ba34fa097ab94b9436e3a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f380b9f34b2e4beeadb0347c00ff3b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35543 } } peers { permanent_uuid: "a43c60b76ba34fa097ab94b9436e3a7e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45849 } } peers { permanent_uuid: "87b20570250544958669924a06841bc7" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 38881 } }
I20260807 02:21:46.725219 5895 heartbeater.cc:503] Master 127.4.221.126:38471 was elected leader, sending a full tablet report...
I20260807 02:21:46.725965 6051 ts_tablet_manager.cc:1428] T b247c36a730649268434e0da43c490de P a43c60b76ba34fa097ab94b9436e3a7e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 02:21:46.726177 6052 ts_tablet_manager.cc:1428] T b247c36a730649268434e0da43c490de P f380b9f34b2e4beeadb0347c00ff3b40: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260807 02:21:46.726265 6034 heartbeater.cc:503] Master 127.4.221.126:38471 was elected leader, sending a full tablet report...
I20260807 02:21:46.726500 5758 heartbeater.cc:503] Master 127.4.221.126:38471 was elected leader, sending a full tablet report...
I20260807 02:21:46.751549 6057 raft_consensus.cc:482] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:21:46.751820 6057 raft_consensus.cc:504] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f380b9f34b2e4beeadb0347c00ff3b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35543 } } peers { permanent_uuid: "a43c60b76ba34fa097ab94b9436e3a7e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45849 } } peers { permanent_uuid: "87b20570250544958669924a06841bc7" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 38881 } }
I20260807 02:21:46.753073 6057 leader_election.cc:288] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f380b9f34b2e4beeadb0347c00ff3b40 (127.4.221.65:35543), a43c60b76ba34fa097ab94b9436e3a7e (127.4.221.67:45849)
I20260807 02:21:46.757507 6047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.753558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:42507 (local address 127.4.221.65:35543)
0807 02:21:46.753695 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:46.753699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:46.753715 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:21:46.753891 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:46.753899 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:46.753946 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:46.754026 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:46.754313 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.754907 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.756331 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.756912 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.757006 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.757049 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:46.757276 (+ 227us) server_negotiation.cc:299] Negotiation successful
0807 02:21:46.757371 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 02:21:46.757511 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.753412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:35543 (local address 127.4.221.66:42507)
0807 02:21:46.753586 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:46.753605 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:46.753716 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:46.754051 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:46.754054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:46.754071 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:46.754195 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.754200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.755020 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.755025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.756193 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.756203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.757033 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.757037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.757147 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.757174 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 02:21:46.757266 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 02:21:46.757353 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":23}
I20260807 02:21:46.757797 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.753412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:45849 (local address 127.4.221.66:44821)
0807 02:21:46.754003 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:46.754015 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:46.754152 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:46.754434 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:46.754437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:46.754453 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:46.754573 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.754579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.755396 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.755402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.756556 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:46.756567 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.757374 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.757377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:46.757471 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.757493 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 02:21:46.757575 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 02:21:46.757644 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":159,"thread_start_us":192,"threads_started":1}
I20260807 02:21:46.757879 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.753548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:44821 (local address 127.4.221.67:45849)
0807 02:21:46.753686 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:46.753691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:46.754102 (+ 411us) server_negotiation.cc:407] Connection header received
0807 02:21:46.754267 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:46.754270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:46.754327 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:46.754411 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:46.754714 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.755283 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.756726 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.757251 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.757363 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.757408 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:46.757608 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 02:21:46.757696 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:21:46.758471 5985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b247c36a730649268434e0da43c490de" candidate_uuid: "87b20570250544958669924a06841bc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a43c60b76ba34fa097ab94b9436e3a7e" is_pre_election: true
I20260807 02:21:46.758471 5712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b247c36a730649268434e0da43c490de" candidate_uuid: "87b20570250544958669924a06841bc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f380b9f34b2e4beeadb0347c00ff3b40" is_pre_election: true
I20260807 02:21:46.758880 5712 raft_consensus.cc:2454] T b247c36a730649268434e0da43c490de P f380b9f34b2e4beeadb0347c00ff3b40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87b20570250544958669924a06841bc7 in term 0.
I20260807 02:21:46.759390 5779 leader_election.cc:302] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87b20570250544958669924a06841bc7, f380b9f34b2e4beeadb0347c00ff3b40; no voters:
I20260807 02:21:46.759657 6057 raft_consensus.cc:2781] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:21:46.759764 6057 raft_consensus.cc:482] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:21:46.759825 6057 raft_consensus.cc:3037] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:46.761198 6057 raft_consensus.cc:504] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f380b9f34b2e4beeadb0347c00ff3b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35543 } } peers { permanent_uuid: "a43c60b76ba34fa097ab94b9436e3a7e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45849 } } peers { permanent_uuid: "87b20570250544958669924a06841bc7" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 38881 } }
I20260807 02:21:46.761497 5985 raft_consensus.cc:2454] T b247c36a730649268434e0da43c490de P a43c60b76ba34fa097ab94b9436e3a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87b20570250544958669924a06841bc7 in term 0.
I20260807 02:21:46.761615 6057 leader_election.cc:288] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [CANDIDATE]: Term 1 election: Requested vote from peers f380b9f34b2e4beeadb0347c00ff3b40 (127.4.221.65:35543), a43c60b76ba34fa097ab94b9436e3a7e (127.4.221.67:45849)
I20260807 02:21:46.762058 5712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b247c36a730649268434e0da43c490de" candidate_uuid: "87b20570250544958669924a06841bc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f380b9f34b2e4beeadb0347c00ff3b40"
I20260807 02:21:46.762080 5984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b247c36a730649268434e0da43c490de" candidate_uuid: "87b20570250544958669924a06841bc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a43c60b76ba34fa097ab94b9436e3a7e"
I20260807 02:21:46.762197 5712 raft_consensus.cc:3037] T b247c36a730649268434e0da43c490de P f380b9f34b2e4beeadb0347c00ff3b40 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:46.762216 5984 raft_consensus.cc:3037] T b247c36a730649268434e0da43c490de P a43c60b76ba34fa097ab94b9436e3a7e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:46.763339 5984 raft_consensus.cc:2454] T b247c36a730649268434e0da43c490de P a43c60b76ba34fa097ab94b9436e3a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87b20570250544958669924a06841bc7 in term 1.
I20260807 02:21:46.763432 5712 raft_consensus.cc:2454] T b247c36a730649268434e0da43c490de P f380b9f34b2e4beeadb0347c00ff3b40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87b20570250544958669924a06841bc7 in term 1.
I20260807 02:21:46.763690 5782 leader_election.cc:302] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87b20570250544958669924a06841bc7, a43c60b76ba34fa097ab94b9436e3a7e; no voters:
I20260807 02:21:46.763880 6057 raft_consensus.cc:2781] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:21:46.764140 6057 raft_consensus.cc:686] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [term 1 LEADER]: Becoming Leader. State: Replica: 87b20570250544958669924a06841bc7, State: Running, Role: LEADER
I20260807 02:21:46.764480 6057 consensus_queue.cc:237] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f380b9f34b2e4beeadb0347c00ff3b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35543 } } peers { permanent_uuid: "a43c60b76ba34fa097ab94b9436e3a7e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45849 } } peers { permanent_uuid: "87b20570250544958669924a06841bc7" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 38881 } }
I20260807 02:21:46.768260 5562 catalog_manager.cc:5495] T b247c36a730649268434e0da43c490de P 87b20570250544958669924a06841bc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87b20570250544958669924a06841bc7 (127.4.221.66). New cstate: current_term: 1 leader_uuid: "87b20570250544958669924a06841bc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f380b9f34b2e4beeadb0347c00ff3b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a43c60b76ba34fa097ab94b9436e3a7e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87b20570250544958669924a06841bc7" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 38881 } health_report { overall_health: HEALTHY } } }
Aug 07 02:21:46 dist-test-slave-69rg krb5kdc[5518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.65@KRBTEST.COM
I20260807 02:21:46.783353 6047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.773553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42728 (local address 127.4.221.65:35543)
0807 02:21:46.773741 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:46.773745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:46.773893 (+ 148us) server_negotiation.cc:407] Connection header received
0807 02:21:46.774002 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:46.774004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:46.774048 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:46.774118 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:46.774860 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.775429 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.776368 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.776613 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.776702 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.779385 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:46.779401 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:46.779418 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:46.779451 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:46.781319 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:46.781771 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:46.781776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:46.781777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:46.781919 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:46.782168 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:46.782170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:46.782171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:46.782459 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:46.782624 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:46.782936 (+ 312us) server_negotiation.cc:299] Negotiation successful
0807 02:21:46.783183 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
Aug 07 02:21:46 dist-test-slave-69rg krb5kdc[5518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069306, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 02:21:46 dist-test-slave-69rg krb5kdc[5518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069306, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.221.65@KRBTEST.COM
I20260807 02:21:46.816901 6047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.807240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.4.221.65:35543)
0807 02:21:46.807404 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:46.807411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:46.807611 (+ 200us) server_negotiation.cc:407] Connection header received
0807 02:21:46.807719 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:46.807721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:46.807767 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:46.807839 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:46.808562 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.809117 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.810107 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.810338 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.810429 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.813159 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:46.813163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:46.813166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:46.813206 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:46.814988 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:46.815489 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:46.815493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:46.815495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:46.815588 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:46.815893 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:46.815897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:46.815899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:46.816097 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:46.816238 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:46.816572 (+ 334us) server_negotiation.cc:299] Negotiation successful
0807 02:21:46.816711 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Aug 07 02:21:46 dist-test-slave-69rg krb5kdc[5518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 02:21:46 dist-test-slave-69rg krb5kdc[5518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.65@KRBTEST.COM
I20260807 02:21:46.845834 6047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:46.837043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42760 (local address 127.4.221.65:35543)
0807 02:21:46.837260 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:46.837264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:46.837328 (+ 64us) server_negotiation.cc:407] Connection header received
0807 02:21:46.837451 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:46.837454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:46.837501 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:46.837575 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:46.838359 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.838880 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.839705 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:46.839920 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:46.840001 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:46.842743 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:46.842746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:46.842748 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:46.842779 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:46.844151 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:46.844585 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:46.844588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:46.844589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:46.844679 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:46.844949 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:46.844952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:46.844953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:46.845145 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:46.845249 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:46.845473 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 02:21:46.845591 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
W20260807 02:21:46.847656 5670 tablet_service.cc:2908] Rejecting scan request for tablet b247c36a730649268434e0da43c490de: Uninitialized: clean time has not yet been initialized
I20260807 02:21:46.850152 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 5627
I20260807 02:21:46.857913 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 5763
I20260807 02:21:46.864902 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 5900
I20260807 02:21:46.871630 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 5531
2026-08-07T02:21:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2495 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:21:46 dist-test-slave-69rg krb5kdc[6075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:21:46 dist-test-slave-69rg krb5kdc[6075](info): set up 2 sockets
Aug 07 02:21:46 dist-test-slave-69rg krb5kdc[6075](info): commencing operation
krb5kdc: starting...
W20260807 02:21:47.961354 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:21:48 dist-test-slave-69rg krb5kdc[6075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:21:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:21:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:21:48.103423 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:46163
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:46163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:48.221581 6088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:48.222008 6088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:48.222127 6088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:48.226643 6088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:21:48.226770 6088 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 02:21:48.226799 6088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:48.226830 6088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:48.226871 6088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:21:48.226907 6088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:21:48.232954 6088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:46163
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:46163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:48.234262 6088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:48.235409 6088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:48.244879 6096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:48.244870 6094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:48.244870 6093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:48.245308 6088 server_base.cc:1034] running on GCE node
I20260807 02:21:48.246011 6088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:48.247196 6088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:48.248414 6088 hybrid_clock.cc:648] HybridClock initialized: now 1786069308248378 us; error 50 us; skew 500 ppm
Aug 07 02:21:48 dist-test-slave-69rg krb5kdc[6075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:48.251573 6088 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:21:48.252502 6088 webserver.cc:460] Webserver started at http://127.4.221.126:38001/ using document root <none> and password file <none>
I20260807 02:21:48.253046 6088 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:48.253108 6088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:48.253293 6088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:48.255548 6088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "fcfc9dea13804b999d155e0782244a98"
format_stamp: "Formatted at 2026-08-07 02:21:48 on dist-test-slave-69rg"
server_key: "1d1207864a76fe25bcc52636b3ace0a3"
server_key_iv: "3a99ee955fe39f066d52030a036fba35"
server_key_version: "encryptionkey@0"
I20260807 02:21:48.256116 6088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "fcfc9dea13804b999d155e0782244a98"
format_stamp: "Formatted at 2026-08-07 02:21:48 on dist-test-slave-69rg"
server_key: "1d1207864a76fe25bcc52636b3ace0a3"
server_key_iv: "3a99ee955fe39f066d52030a036fba35"
server_key_version: "encryptionkey@0"
I20260807 02:21:48.260038 6088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:21:48.262678 6103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:48.263927 6088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:21:48.264088 6088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "fcfc9dea13804b999d155e0782244a98"
format_stamp: "Formatted at 2026-08-07 02:21:48 on dist-test-slave-69rg"
server_key: "1d1207864a76fe25bcc52636b3ace0a3"
server_key_iv: "3a99ee955fe39f066d52030a036fba35"
server_key_version: "encryptionkey@0"
I20260807 02:21:48.264214 6088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:48.277071 6088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:48.280486 6088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:48.280722 6088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:48.289865 6088 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:46163
I20260807 02:21:48.289884 6167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:46163 every 8 connection(s)
I20260807 02:21:48.290907 6088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:21:48.291213 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 6088
I20260807 02:21:48.291350 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:21:48.291674 4981 external_mini_cluster.cc:1428] Setting key 37382dac605cd40f96ef0c1c9986ca89
I20260807 02:21:48.294319 6168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:48.300213 6168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98: Bootstrap starting.
Aug 07 02:21:48 dist-test-slave-69rg krb5kdc[6075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:48.302814 6168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:48.304196 6168 log.cc:824] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98: Log is configured to *not* fsync() on all Append() calls
I20260807 02:21:48.306428 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:48.293225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49746 (local address 127.4.221.126:46163)
0807 02:21:48.293675 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:48.293690 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:48.293726 (+ 36us) server_negotiation.cc:407] Connection header received
0807 02:21:48.294525 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:48.294544 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:48.294873 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:48.295215 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:48.296085 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.297113 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.298051 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.298544 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.298723 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:48.301464 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:48.301469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:48.301483 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:48.301513 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:48.303361 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:48.303846 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:48.303852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:48.303856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:48.303972 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:48.304268 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:48.304272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:48.304274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:48.304753 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:48.304966 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:48.305300 (+ 334us) server_negotiation.cc:299] Negotiation successful
0807 02:21:48.305598 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":145,"threads_started":1}
I20260807 02:21:48.306434 6168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98: No bootstrap required, opened a new log
I20260807 02:21:48.309708 6168 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfc9dea13804b999d155e0782244a98" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46163 } }
I20260807 02:21:48.310031 6168 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:48.310127 6168 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfc9dea13804b999d155e0782244a98, State: Initialized, Role: FOLLOWER
I20260807 02:21:48.310606 6168 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfc9dea13804b999d155e0782244a98" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46163 } }
I20260807 02:21:48.310761 6168 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:21:48.310829 6168 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:21:48.310926 6168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:48.311973 6168 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfc9dea13804b999d155e0782244a98" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46163 } }
I20260807 02:21:48.312323 6168 leader_election.cc:302] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcfc9dea13804b999d155e0782244a98; no voters:
I20260807 02:21:48.312666 6168 leader_election.cc:288] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:21:48.312806 6174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:21:48.313071 6174 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [term 1 LEADER]: Becoming Leader. State: Replica: fcfc9dea13804b999d155e0782244a98, State: Running, Role: LEADER
I20260807 02:21:48.313406 6174 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfc9dea13804b999d155e0782244a98" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46163 } }
I20260807 02:21:48.314051 6168 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:21:48.314715 6176 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcfc9dea13804b999d155e0782244a98. Latest consensus state: current_term: 1 leader_uuid: "fcfc9dea13804b999d155e0782244a98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfc9dea13804b999d155e0782244a98" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46163 } } }
I20260807 02:21:48.314858 6176 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [sys.catalog]: This master's current role is: LEADER
I20260807 02:21:48.315171 6175 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcfc9dea13804b999d155e0782244a98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcfc9dea13804b999d155e0782244a98" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46163 } } }
I20260807 02:21:48.315277 6175 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98 [sys.catalog]: This master's current role is: LEADER
I20260807 02:21:48.315513 6181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:21:48.318943 6181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:21:48.325122 6181 catalog_manager.cc:1347] Generated new cluster ID: a90173faa71b47b984e0a8ad466b5510
I20260807 02:21:48.325207 6181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:21:48.340502 6181 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:21:48.341549 6181 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:21:48.350770 6181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98: Generated new TSK 0
I20260807 02:21:48.351567 6181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:21:48.385648 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:46163
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:48.506412 6195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:48.506744 6195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:48.506837 6195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:48.510991 6195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:48.511099 6195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:48.511176 6195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:21:48.511260 6195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:21:48.517237 6195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.221.126:46163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:48.518530 6195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:48.519608 6195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:48.529409 6201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:48.529410 6200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:48.529410 6203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:48.529583 6195 server_base.cc:1034] running on GCE node
I20260807 02:21:48.530186 6195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:48.530907 6195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:48.532080 6195 hybrid_clock.cc:648] HybridClock initialized: now 1786069308532054 us; error 45 us; skew 500 ppm
Aug 07 02:21:48 dist-test-slave-69rg krb5kdc[6075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:48.535377 6195 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:21:48.536463 6195 webserver.cc:460] Webserver started at http://127.4.221.65:34095/ using document root <none> and password file <none>
I20260807 02:21:48.537091 6195 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:48.537194 6195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:48.537456 6195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:48.539746 6195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "1b79e3d055c341d0be5bc3d10ebaf235"
format_stamp: "Formatted at 2026-08-07 02:21:48 on dist-test-slave-69rg"
server_key: "89dff7f6e69e14520eece339950c5bc2"
server_key_iv: "e594d016590abb90679bc56b69d8fc65"
server_key_version: "encryptionkey@0"
I20260807 02:21:48.540362 6195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "1b79e3d055c341d0be5bc3d10ebaf235"
format_stamp: "Formatted at 2026-08-07 02:21:48 on dist-test-slave-69rg"
server_key: "89dff7f6e69e14520eece339950c5bc2"
server_key_iv: "e594d016590abb90679bc56b69d8fc65"
server_key_version: "encryptionkey@0"
I20260807 02:21:48.544353 6195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:21:48.546993 6210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:48.548245 6195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:21:48.548414 6195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "1b79e3d055c341d0be5bc3d10ebaf235"
format_stamp: "Formatted at 2026-08-07 02:21:48 on dist-test-slave-69rg"
server_key: "89dff7f6e69e14520eece339950c5bc2"
server_key_iv: "e594d016590abb90679bc56b69d8fc65"
server_key_version: "encryptionkey@0"
I20260807 02:21:48.548542 6195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:48.564235 6195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:48.567555 6195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:48.567804 6195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:48.569561 6195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:21:48.569643 6195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:48.569715 6195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:21:48.569752 6195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:21:48 dist-test-slave-69rg krb5kdc[6075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:48.581770 6195 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:39137
I20260807 02:21:48.581924 6328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:39137 every 8 connection(s)
I20260807 02:21:48.583009 6195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:21:48.584247 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 6195
I20260807 02:21:48.584378 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:21:48.584684 4981 external_mini_cluster.cc:1428] Setting key a3f5dddcccb43e7824c6c913bf2671e8
I20260807 02:21:48.585803 6219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:48.571115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46163 (local address 127.4.221.65:37771)
0807 02:21:48.571668 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:48.571716 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:48.572663 (+ 947us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:48.573497 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:48.573503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:48.573976 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:48.574483 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:48.574497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.575434 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.575438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:48.576492 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:48.576502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.577120 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.577128 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:48.577389 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:48.578227 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:48.578245 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:48.580380 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:48.582611 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:48.582617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:48.582628 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:48.582906 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:48.583268 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:48.583271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:48.583272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:48.583416 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:48.583895 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:48.583902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:48.584235 (+ 333us) client_negotiation.cc:769] Sending connection context
0807 02:21:48.584475 (+ 240us) client_negotiation.cc:240] Negotiation successful
0807 02:21:48.584847 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":148,"threads_started":1}
I20260807 02:21:48.586375 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:48.570520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:37771 (local address 127.4.221.126:46163)
0807 02:21:48.570664 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:48.570668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:48.571834 (+ 1166us) server_negotiation.cc:407] Connection header received
0807 02:21:48.572926 (+ 1092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:48.572931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:48.573001 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:48.573091 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:48.574650 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.575283 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.576634 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.576889 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.577033 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:48.580634 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:48.580639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:48.580642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:48.580682 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:48.582458 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:48.583029 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:48.583033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:48.583036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:48.583142 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:48.583533 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:48.583537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:48.583538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:48.583774 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:48.583898 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:48.586021 (+ 2123us) server_negotiation.cc:299] Negotiation successful
0807 02:21:48.586175 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 02:21:48.587464 6329 heartbeater.cc:347] Connected to a master server at 127.4.221.126:46163
I20260807 02:21:48.587633 6329 heartbeater.cc:464] Registering TS with master...
I20260807 02:21:48.588251 6329 heartbeater.cc:511] Master 127.4.221.126:46163 requested a full tablet report, sending...
I20260807 02:21:48.590124 6120 ts_manager.cc:194] Registered new tserver with Master: 1b79e3d055c341d0be5bc3d10ebaf235 (127.4.221.65:39137)
I20260807 02:21:48.591284 6120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:37771
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
I20260807 02:21:48.601538 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:48.589931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:47811 (local address 127.4.221.126:46163)
0807 02:21:48.590048 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:48.590052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:48.594146 (+ 4094us) server_negotiation.cc:407] Connection header received
0807 02:21:48.594312 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:48.594315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:48.594356 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:48.594441 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:48.594908 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.595466 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.596423 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.596651 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.596764 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:48.597919 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:48.597926 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:48.597932 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:48.597964 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:48.599708 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:48.600160 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:48.600167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:48.600171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:48.600267 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:48.600536 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:48.600542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:48.600547 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:48.600824 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:48.600944 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:48.601295 (+ 351us) server_negotiation.cc:299] Negotiation successful
0807 02:21:48.601408 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 02:21:48.601684 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:48.589780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46163 (local address 127.4.221.65:47811)
0807 02:21:48.594046 (+ 4266us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:48.594061 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:48.594192 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:48.594461 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:48.594464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:48.594686 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:48.594788 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:48.594795 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.595596 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.595599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:48.596304 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:48.596313 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.596774 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.596777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:48.596873 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:48.597389 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:48.597401 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:48.597743 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:48.599831 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:48.599834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:48.599836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:48.600044 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:48.600363 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:48.600365 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:48.600367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:48.600445 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:48.600967 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:48.600971 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:48.601107 (+ 136us) client_negotiation.cc:769] Sending connection context
0807 02:21:48.601231 (+ 124us) client_negotiation.cc:240] Negotiation successful
0807 02:21:48.601413 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4179,"thread_start_us":140,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:21:48.618537 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:46163
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:48.736946 6338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:48.737296 6338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:48.737387 6338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:48.741572 6338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:48.741698 6338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:48.741778 6338 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:21:48.741899 6338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:21:48.747682 6338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.221.126:46163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:48.748942 6338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:48.750074 6338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:48.760476 6344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:48.760502 6343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:48.760495 6338 server_base.cc:1034] running on GCE node
W20260807 02:21:48.760515 6346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:48.761250 6338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:48.762065 6338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:48.763283 6338 hybrid_clock.cc:648] HybridClock initialized: now 1786069308763259 us; error 35 us; skew 500 ppm
Aug 07 02:21:48 dist-test-slave-69rg krb5kdc[6075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:48.766430 6338 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:21:48.767480 6338 webserver.cc:460] Webserver started at http://127.4.221.66:44855/ using document root <none> and password file <none>
I20260807 02:21:48.768096 6338 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:48.768195 6338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:48.768481 6338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:48.770869 6338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "2ab59712909c4b988ae8b196578c3c05"
format_stamp: "Formatted at 2026-08-07 02:21:48 on dist-test-slave-69rg"
server_key: "3f736e49dea12fc692bd7137f9974754"
server_key_iv: "6f18684f7beab7dc5a57a0262f04fc9a"
server_key_version: "encryptionkey@0"
I20260807 02:21:48.771512 6338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "2ab59712909c4b988ae8b196578c3c05"
format_stamp: "Formatted at 2026-08-07 02:21:48 on dist-test-slave-69rg"
server_key: "3f736e49dea12fc692bd7137f9974754"
server_key_iv: "6f18684f7beab7dc5a57a0262f04fc9a"
server_key_version: "encryptionkey@0"
I20260807 02:21:48.775660 6338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260807 02:21:48.778239 6353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:48.779528 6338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:21:48.779677 6338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "2ab59712909c4b988ae8b196578c3c05"
format_stamp: "Formatted at 2026-08-07 02:21:48 on dist-test-slave-69rg"
server_key: "3f736e49dea12fc692bd7137f9974754"
server_key_iv: "6f18684f7beab7dc5a57a0262f04fc9a"
server_key_version: "encryptionkey@0"
I20260807 02:21:48.779805 6338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:48.794610 6338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:48.798097 6338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:48.798334 6338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:48.799918 6338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:21:48.799996 6338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:48.800076 6338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:21:48.800118 6338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:21:48 dist-test-slave-69rg krb5kdc[6075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:48.811909 6338 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:36427
I20260807 02:21:48.812191 6470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:36427 every 8 connection(s)
I20260807 02:21:48.812906 6338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:21:48.815145 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:48.800839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:41653 (local address 127.4.221.126:46163)
0807 02:21:48.800980 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:48.800984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:48.802171 (+ 1187us) server_negotiation.cc:407] Connection header received
0807 02:21:48.803396 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:48.803403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:48.803481 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:48.803589 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:48.805063 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.805986 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.807211 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.807493 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.807606 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:48.810851 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:48.810854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:48.810856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:48.810885 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:48.812637 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:48.813218 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:48.813220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:48.813222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:48.813300 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:48.813777 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:48.813781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:48.813782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:48.814043 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:48.814123 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:48.814769 (+ 646us) server_negotiation.cc:299] Negotiation successful
0807 02:21:48.814921 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 02:21:48.815868 6362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:48.801399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46163 (local address 127.4.221.66:41653)
0807 02:21:48.802001 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:48.802049 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:48.803032 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:48.803829 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:48.803836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:48.804315 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:48.804902 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:48.804919 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.806106 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.806110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:48.807034 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:48.807044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.807666 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.807671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:48.807850 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:48.808528 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:48.808545 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:48.810634 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:48.812755 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:48.812760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:48.812773 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:48.813117 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:48.813436 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:48.813439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:48.813441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:48.813644 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:48.814122 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:48.814127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:48.814355 (+ 228us) client_negotiation.cc:769] Sending connection context
0807 02:21:48.814582 (+ 227us) client_negotiation.cc:240] Negotiation successful
0807 02:21:48.814952 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":146,"threads_started":1}
I20260807 02:21:48.816301 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 6338
I20260807 02:21:48.816396 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:21:48.816615 4981 external_mini_cluster.cc:1428] Setting key 15594463f48b05ecb8975b1dd3bd6d7e
I20260807 02:21:48.818372 6471 heartbeater.cc:347] Connected to a master server at 127.4.221.126:46163
I20260807 02:21:48.818531 6471 heartbeater.cc:464] Registering TS with master...
I20260807 02:21:48.819140 6471 heartbeater.cc:511] Master 127.4.221.126:46163 requested a full tablet report, sending...
I20260807 02:21:48.820521 6120 ts_manager.cc:194] Registered new tserver with Master: 2ab59712909c4b988ae8b196578c3c05 (127.4.221.66:36427)
I20260807 02:21:48.821126 6120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:41653
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
I20260807 02:21:48.829975 6478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:48.821948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46163 (local address 127.4.221.66:44833)
0807 02:21:48.822251 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:48.822267 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:48.822413 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:48.822766 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:48.822768 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:48.823014 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:48.823133 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:48.823141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.824015 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.824017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:48.824575 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:48.824582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.825323 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.825329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:48.825439 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:48.826066 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:48.826087 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:48.826444 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:48.828425 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:48.828428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:48.828429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:48.828628 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:48.828890 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:48.828892 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:48.828893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:48.828968 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:48.829434 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:48.829437 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:48.829523 (+ 86us) client_negotiation.cc:769] Sending connection context
0807 02:21:48.829608 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 02:21:48.829730 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":106,"threads_started":1}
I20260807 02:21:48.829982 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:48.821783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:44833 (local address 127.4.221.126:46163)
0807 02:21:48.821951 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:48.821956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:48.822338 (+ 382us) server_negotiation.cc:407] Connection header received
0807 02:21:48.822524 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:48.822528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:48.822576 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:48.822629 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:48.823273 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.823892 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.824700 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:48.825141 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:48.825272 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:48.826600 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:48.826606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:48.826612 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:48.826645 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:48.828283 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:48.828719 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:48.828722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:48.828723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:48.828808 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:48.829049 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:48.829051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:48.829052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:48.829360 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:48.829442 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:48.829674 (+ 232us) server_negotiation.cc:299] Negotiation successful
0807 02:21:48.829776 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:21:48.849220 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:46163
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:48.968057 6480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:48.968361 6480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:48.968410 6480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:48.972723 6480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:48.972803 6480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:48.972846 6480 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:21:48.972882 6480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:21:48.978488 6480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.221.126:46163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:48.979609 6480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:48.980643 6480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:48.991007 6486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:48.991082 6480 server_base.cc:1034] running on GCE node
W20260807 02:21:48.991000 6485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:48.991007 6488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:48.991676 6480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:48.992312 6480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:48.993551 6480 hybrid_clock.cc:648] HybridClock initialized: now 1786069308993515 us; error 44 us; skew 500 ppm
Aug 07 02:21:48 dist-test-slave-69rg krb5kdc[6075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:48.996877 6480 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:21:48.998039 6480 webserver.cc:460] Webserver started at http://127.4.221.67:35853/ using document root <none> and password file <none>
I20260807 02:21:48.998653 6480 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:48.998754 6480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:48.999007 6480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:49.001206 6480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "1dfff0061e864417abc538ed2df2930b"
format_stamp: "Formatted at 2026-08-07 02:21:48 on dist-test-slave-69rg"
server_key: "2a1feea54dfad8582784b611e0e2d56b"
server_key_iv: "1b27c8a39c2f2a706f29cf22142ea4cc"
server_key_version: "encryptionkey@0"
I20260807 02:21:49.001870 6480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "1dfff0061e864417abc538ed2df2930b"
format_stamp: "Formatted at 2026-08-07 02:21:48 on dist-test-slave-69rg"
server_key: "2a1feea54dfad8582784b611e0e2d56b"
server_key_iv: "1b27c8a39c2f2a706f29cf22142ea4cc"
server_key_version: "encryptionkey@0"
I20260807 02:21:49.006008 6480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 02:21:49.008652 6495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:49.009948 6480 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:21:49.010090 6480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "1dfff0061e864417abc538ed2df2930b"
format_stamp: "Formatted at 2026-08-07 02:21:48 on dist-test-slave-69rg"
server_key: "2a1feea54dfad8582784b611e0e2d56b"
server_key_iv: "1b27c8a39c2f2a706f29cf22142ea4cc"
server_key_version: "encryptionkey@0"
I20260807 02:21:49.010243 6480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:49.036388 6480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:49.039656 6480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:49.039891 6480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:49.041510 6480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:21:49.041589 6480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:49.041652 6480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:21:49.041705 6480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:21:49 dist-test-slave-69rg krb5kdc[6075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:49.053570 6480 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:44527
I20260807 02:21:49.053654 6610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:44527 every 8 connection(s)
I20260807 02:21:49.054615 6480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:21:49.056653 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.042347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:56017 (local address 127.4.221.126:46163)
0807 02:21:49.042528 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.042533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.043552 (+ 1019us) server_negotiation.cc:407] Connection header received
0807 02:21:49.044850 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.044856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.044932 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.045042 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:49.046919 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.047606 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.048808 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.049158 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.049282 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.052472 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:49.052476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:49.052478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:49.052535 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:49.054356 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:49.054980 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:49.054983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:49.054985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:49.055110 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:49.055501 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:49.055505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:49.055507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:49.055718 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:49.055840 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.056340 (+ 500us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.056478 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 02:21:49.057451 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.042841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46163 (local address 127.4.221.67:56017)
0807 02:21:49.043349 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:49.043412 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:49.044583 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:49.045329 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:49.045340 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:49.045948 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:49.046730 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.046754 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.047746 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.047750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.048666 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.048676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.049342 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.049347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.049533 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.050239 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:49.050255 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:49.052264 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:49.054494 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:49.054499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:49.054515 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:49.054869 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:49.055224 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:49.055227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:49.055229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:49.055411 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:49.055838 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:49.055845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:49.056100 (+ 255us) client_negotiation.cc:769] Sending connection context
0807 02:21:49.056302 (+ 202us) client_negotiation.cc:240] Negotiation successful
0807 02:21:49.056628 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":114,"threads_started":1}
I20260807 02:21:49.057525 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 6480
I20260807 02:21:49.057654 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:21:49.058014 4981 external_mini_cluster.cc:1428] Setting key 0035c48f67d0f2720dae9c3bcac8ff41
I20260807 02:21:49.058902 6611 heartbeater.cc:347] Connected to a master server at 127.4.221.126:46163
I20260807 02:21:49.059216 6611 heartbeater.cc:464] Registering TS with master...
I20260807 02:21:49.059831 6611 heartbeater.cc:511] Master 127.4.221.126:46163 requested a full tablet report, sending...
I20260807 02:21:49.060881 6119 ts_manager.cc:194] Registered new tserver with Master: 1dfff0061e864417abc538ed2df2930b (127.4.221.67:44527)
I20260807 02:21:49.061714 6119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:56017
I20260807 02:21:49.068564 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.061331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:58725 (local address 127.4.221.126:46163)
0807 02:21:49.061480 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.061485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.061707 (+ 222us) server_negotiation.cc:407] Connection header received
0807 02:21:49.061892 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.061895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.061948 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.062035 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:49.062679 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.063180 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.064226 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.064462 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.064546 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.065709 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:49.065712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:49.065714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:49.065734 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:49.067028 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:49.067469 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:49.067471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:49.067472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:49.067546 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:49.067800 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:49.067802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:49.067803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:49.067969 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:49.068046 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.068346 (+ 300us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.068448 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 02:21:49.068578 6617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.061293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46163 (local address 127.4.221.67:58725)
0807 02:21:49.061618 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:49.061633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:49.061740 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:49.062164 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:49.062168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:49.062434 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:49.062554 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.062562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.063308 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.063312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.064113 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.064123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.064564 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.064566 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.064640 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.065223 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:49.065236 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:49.065575 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:49.067159 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:49.067162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:49.067163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:49.067368 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:49.067628 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:49.067630 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:49.067632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:49.067719 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:49.068048 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:49.068050 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:49.068145 (+ 95us) client_negotiation.cc:769] Sending connection context
0807 02:21:49.068279 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 02:21:49.068405 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":133,"threads_started":1}
I20260807 02:21:49.072158 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 02:21:49 dist-test-slave-69rg krb5kdc[6075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 02:21:49 dist-test-slave-69rg krb5kdc[6075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:49.100734 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.091441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49762 (local address 127.4.221.126:46163)
0807 02:21:49.091594 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.091599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.091613 (+ 14us) server_negotiation.cc:407] Connection header received
0807 02:21:49.091660 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.091662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.091706 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.091785 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:49.092536 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.093075 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.093995 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.094216 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.094307 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.097196 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:49.097200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:49.097202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:49.097236 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:49.098913 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:49.099387 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:49.099390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:49.099392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:49.099482 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:49.099791 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:49.099794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:49.099796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:49.100002 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:49.100162 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.100416 (+ 254us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.100542 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 02:21:49.103983 6119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:21:49.107101 6119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:21:49.121212 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.115727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44316 (local address 127.4.221.66:36427)
0807 02:21:49.116185 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.116197 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.116223 (+ 26us) server_negotiation.cc:407] Connection header received
0807 02:21:49.116425 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.116431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.116645 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.116805 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:49.117337 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.118403 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.119913 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.120471 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.120593 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.120767 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.120951 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.121049 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":127,"threads_started":1}
I20260807 02:21:49.121210 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.115574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:36427 (local address 127.0.0.1:44316)
0807 02:21:49.116001 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:49.116062 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:49.116206 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:49.116996 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:49.117002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:49.117028 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:49.117188 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.117196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.118529 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.118534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.119768 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.119780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.120605 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.120609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.120721 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.120775 (+ 54us) client_negotiation.cc:769] Sending connection context
0807 02:21:49.120920 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 02:21:49.121049 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":128,"threads_started":1}
I20260807 02:21:49.122434 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.116447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.4.221.65:39137)
0807 02:21:49.116836 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.116845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.117199 (+ 354us) server_negotiation.cc:407] Connection header received
0807 02:21:49.117427 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.117431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.117569 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.117766 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:49.118067 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.119226 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.120698 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.121766 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.121914 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.122055 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.122155 (+ 100us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.122251 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":81,"threads_started":1}
I20260807 02:21:49.122774 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.115922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:39137 (local address 127.0.0.1:47030)
0807 02:21:49.117090 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:49.117104 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:49.117282 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:49.117759 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:49.117763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:49.117782 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:49.117947 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.117954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.119370 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.119373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.120556 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.120566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.121916 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.121920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.122012 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.122036 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 02:21:49.122125 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 02:21:49.122205 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"spinlock_wait_cycles":768,"thread_start_us":87,"threads_started":1}
I20260807 02:21:49.124142 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.116384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:44527 (local address 127.0.0.1:36806)
0807 02:21:49.117158 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:49.117172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:49.117275 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:49.118349 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:49.118352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:49.118370 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:49.118457 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.118466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.119824 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.119827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.120929 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.120939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.122817 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.122820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.123808 (+ 988us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.123835 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 02:21:49.123922 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 02:21:49.124004 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"spinlock_wait_cycles":4480,"thread_start_us":94,"threads_started":1}
I20260807 02:21:49.124233 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.116776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.4.221.67:44527)
0807 02:21:49.117225 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.117237 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.117260 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:21:49.117994 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.117999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.118204 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.118364 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:49.118537 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.119687 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.121077 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.122287 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.122387 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.122741 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.123978 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.124083 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":116,"threads_started":1}
I20260807 02:21:49.124541 6404 tablet_service.cc:1467] Processing CreateTablet for tablet 0ddca34d4d8c42e3a60a0e9ac9200366 (DEFAULT_TABLE table=test-table [id=283d4fe82c9940cfa078d86f829443dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:49.125602 6404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ddca34d4d8c42e3a60a0e9ac9200366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:49.125489 6261 tablet_service.cc:1467] Processing CreateTablet for tablet 0ddca34d4d8c42e3a60a0e9ac9200366 (DEFAULT_TABLE table=test-table [id=283d4fe82c9940cfa078d86f829443dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:49.126758 6261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ddca34d4d8c42e3a60a0e9ac9200366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:49.128587 6545 tablet_service.cc:1467] Processing CreateTablet for tablet 0ddca34d4d8c42e3a60a0e9ac9200366 (DEFAULT_TABLE table=test-table [id=283d4fe82c9940cfa078d86f829443dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:49.129595 6545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ddca34d4d8c42e3a60a0e9ac9200366. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:49.132997 6634 tablet_bootstrap.cc:492] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05: Bootstrap starting.
I20260807 02:21:49.134588 6636 tablet_bootstrap.cc:492] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235: Bootstrap starting.
I20260807 02:21:49.135923 6634 tablet_bootstrap.cc:654] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:49.136785 6636 tablet_bootstrap.cc:654] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:49.136842 6634 log.cc:824] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05: Log is configured to *not* fsync() on all Append() calls
I20260807 02:21:49.137429 6637 tablet_bootstrap.cc:492] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b: Bootstrap starting.
I20260807 02:21:49.137642 6636 log.cc:824] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235: Log is configured to *not* fsync() on all Append() calls
I20260807 02:21:49.139032 6634 tablet_bootstrap.cc:492] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05: No bootstrap required, opened a new log
I20260807 02:21:49.139269 6634 ts_tablet_manager.cc:1397] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 02:21:49.139724 6636 tablet_bootstrap.cc:492] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235: No bootstrap required, opened a new log
I20260807 02:21:49.139842 6637 tablet_bootstrap.cc:654] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:49.139947 6636 ts_tablet_manager.cc:1397] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 02:21:49.140687 6637 log.cc:824] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b: Log is configured to *not* fsync() on all Append() calls
I20260807 02:21:49.143157 6637 tablet_bootstrap.cc:492] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b: No bootstrap required, opened a new log
I20260807 02:21:49.143376 6637 ts_tablet_manager.cc:1397] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 02:21:49.143244 6634 raft_consensus.cc:348] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } }
I20260807 02:21:49.143532 6634 raft_consensus.cc:374] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:49.143610 6634 raft_consensus.cc:729] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab59712909c4b988ae8b196578c3c05, State: Initialized, Role: FOLLOWER
I20260807 02:21:49.144089 6634 consensus_queue.cc:260] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } }
I20260807 02:21:49.144079 6636 raft_consensus.cc:348] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } }
I20260807 02:21:49.144321 6636 raft_consensus.cc:374] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:49.144371 6636 raft_consensus.cc:729] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b79e3d055c341d0be5bc3d10ebaf235, State: Initialized, Role: FOLLOWER
I20260807 02:21:49.144761 6636 consensus_queue.cc:260] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } }
I20260807 02:21:49.145186 6634 ts_tablet_manager.cc:1428] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 02:21:49.145486 6329 heartbeater.cc:503] Master 127.4.221.126:46163 was elected leader, sending a full tablet report...
I20260807 02:21:49.145541 6471 heartbeater.cc:503] Master 127.4.221.126:46163 was elected leader, sending a full tablet report...
I20260807 02:21:49.145705 6636 ts_tablet_manager.cc:1428] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 02:21:49.147290 6637 raft_consensus.cc:348] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } }
I20260807 02:21:49.147549 6637 raft_consensus.cc:374] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:49.147610 6637 raft_consensus.cc:729] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dfff0061e864417abc538ed2df2930b, State: Initialized, Role: FOLLOWER
I20260807 02:21:49.148044 6637 consensus_queue.cc:260] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } }
I20260807 02:21:49.149055 6637 ts_tablet_manager.cc:1428] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 02:21:49.149431 6611 heartbeater.cc:503] Master 127.4.221.126:46163 was elected leader, sending a full tablet report...
I20260807 02:21:49.164193 6643 raft_consensus.cc:482] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:21:49.164428 6643 raft_consensus.cc:504] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } }
I20260807 02:21:49.165599 6643 leader_election.cc:288] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ab59712909c4b988ae8b196578c3c05 (127.4.221.66:36427), 1b79e3d055c341d0be5bc3d10ebaf235 (127.4.221.65:39137)
I20260807 02:21:49.170778 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.166110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:51441 (local address 127.4.221.65:39137)
0807 02:21:49.166255 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.166261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.166628 (+ 367us) server_negotiation.cc:407] Connection header received
0807 02:21:49.166715 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.166718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.166788 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.166879 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:49.167304 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.168169 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.169623 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.170110 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.170275 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.170337 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.170548 (+ 211us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.170625 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 02:21:49.170795 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.166225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:56115 (local address 127.4.221.66:36427)
0807 02:21:49.166375 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.166380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.166397 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:21:49.166465 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.166468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.166535 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.166626 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:49.166978 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.167749 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.169637 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.170151 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.170289 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.170349 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.170578 (+ 229us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.170658 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 02:21:49.170825 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.165978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:39137 (local address 127.4.221.67:51441)
0807 02:21:49.166478 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:49.166494 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:49.166626 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:49.166974 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:49.166977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:49.166997 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:49.167129 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.167136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.168316 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.168322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.169477 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.169488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.170258 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.170262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.170372 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.170435 (+ 63us) client_negotiation.cc:769] Sending connection context
0807 02:21:49.170528 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 02:21:49.170617 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":90,"thread_start_us":137,"threads_started":1}
I20260807 02:21:49.170795 6504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.165978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:36427 (local address 127.4.221.67:56115)
0807 02:21:49.166225 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:49.166242 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:49.166356 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:49.166657 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:49.166660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:49.166679 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:49.166818 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.166829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.167865 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.167870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.169498 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.169510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.170282 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.170285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.170412 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.170462 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 02:21:49.170559 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 02:21:49.170653 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":60}
I20260807 02:21:49.171695 6281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddca34d4d8c42e3a60a0e9ac9200366" candidate_uuid: "1dfff0061e864417abc538ed2df2930b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" is_pre_election: true
I20260807 02:21:49.171867 6424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddca34d4d8c42e3a60a0e9ac9200366" candidate_uuid: "1dfff0061e864417abc538ed2df2930b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab59712909c4b988ae8b196578c3c05" is_pre_election: true
I20260807 02:21:49.172072 6281 raft_consensus.cc:2454] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dfff0061e864417abc538ed2df2930b in term 0.
I20260807 02:21:49.172204 6424 raft_consensus.cc:2454] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dfff0061e864417abc538ed2df2930b in term 0.
I20260807 02:21:49.172693 6499 leader_election.cc:302] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b79e3d055c341d0be5bc3d10ebaf235, 1dfff0061e864417abc538ed2df2930b, 2ab59712909c4b988ae8b196578c3c05; no voters:
I20260807 02:21:49.172993 6643 raft_consensus.cc:2781] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:21:49.173095 6643 raft_consensus.cc:482] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:21:49.173161 6643 raft_consensus.cc:3037] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:49.174378 6643 raft_consensus.cc:504] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } }
I20260807 02:21:49.174767 6643 leader_election.cc:288] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [CANDIDATE]: Term 1 election: Requested vote from peers 2ab59712909c4b988ae8b196578c3c05 (127.4.221.66:36427), 1b79e3d055c341d0be5bc3d10ebaf235 (127.4.221.65:39137)
I20260807 02:21:49.175174 6424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddca34d4d8c42e3a60a0e9ac9200366" candidate_uuid: "1dfff0061e864417abc538ed2df2930b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab59712909c4b988ae8b196578c3c05"
I20260807 02:21:49.175325 6424 raft_consensus.cc:3037] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:49.176966 6424 raft_consensus.cc:2454] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dfff0061e864417abc538ed2df2930b in term 1.
I20260807 02:21:49.177423 6498 leader_election.cc:302] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dfff0061e864417abc538ed2df2930b, 2ab59712909c4b988ae8b196578c3c05; no voters:
I20260807 02:21:49.177371 6281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddca34d4d8c42e3a60a0e9ac9200366" candidate_uuid: "1dfff0061e864417abc538ed2df2930b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b79e3d055c341d0be5bc3d10ebaf235"
I20260807 02:21:49.177647 6643 raft_consensus.cc:2781] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:21:49.177616 6281 raft_consensus.cc:3037] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:49.177960 6643 raft_consensus.cc:686] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [term 1 LEADER]: Becoming Leader. State: Replica: 1dfff0061e864417abc538ed2df2930b, State: Running, Role: LEADER
I20260807 02:21:49.178369 6643 consensus_queue.cc:237] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } }
I20260807 02:21:49.178959 6281 raft_consensus.cc:2454] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dfff0061e864417abc538ed2df2930b in term 1.
I20260807 02:21:49.182175 6122 catalog_manager.cc:5495] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dfff0061e864417abc538ed2df2930b (127.4.221.67). New cstate: current_term: 1 leader_uuid: "1dfff0061e864417abc538ed2df2930b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } health_report { overall_health: UNKNOWN } } }
I20260807 02:21:49.215068 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.210503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.4.221.67:44527)
0807 02:21:49.210692 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.210696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.210713 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:21:49.210910 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.210913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.210974 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.211074 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:21:49.211406 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.211945 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.212953 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.213216 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.213315 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.213794 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:21:49.214367 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:21:49.214698 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.214826 (+ 128us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.214894 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 02:21:49.222146 6424 raft_consensus.cc:1261] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05 [term 1 FOLLOWER]: Refusing update from remote peer 1dfff0061e864417abc538ed2df2930b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:49.223007 6643 consensus_queue.cc:1035] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:21:49.224077 6281 raft_consensus.cc:1261] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 1 FOLLOWER]: Refusing update from remote peer 1dfff0061e864417abc538ed2df2930b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:49.224809 6643 consensus_queue.cc:1035] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:21:49.231007 6648 mvcc.cc:205] Tried to move back new op lower bound from 7315739890564411392 to 7315739890398142464. Current Snapshot: MvccSnapshot[applied={T|T < 7315739890564411392}]
I20260807 02:21:49.233776 6651 mvcc.cc:205] Tried to move back new op lower bound from 7315739890564411392 to 7315739890398142464. Current Snapshot: MvccSnapshot[applied={T|T < 7315739890564411392}]
I20260807 02:21:49.246234 6119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:21:49.246511 6119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:21:49.249521 6119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98: Sending DeleteTablet for 3 replicas of tablet 0ddca34d4d8c42e3a60a0e9ac9200366
I20260807 02:21:49.250627 6261 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ddca34d4d8c42e3a60a0e9ac9200366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:49 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:47030
I20260807 02:21:49.250676 6404 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ddca34d4d8c42e3a60a0e9ac9200366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:49 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:44316
I20260807 02:21:49.250875 6545 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ddca34d4d8c42e3a60a0e9ac9200366 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:49 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:36806
I20260807 02:21:49.251370 6659 tablet_replica.cc:330] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b: stopping tablet replica
I20260807 02:21:49.251737 6659 raft_consensus.cc:2229] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:21:49.251814 6657 tablet_replica.cc:330] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05: stopping tablet replica
I20260807 02:21:49.252202 6659 raft_consensus.cc:2258] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:49.252300 6658 tablet_replica.cc:330] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235: stopping tablet replica
I20260807 02:21:49.252218 6119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 02:21:49.252605 6658 raft_consensus.cc:2229] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:21:49.252912 6658 raft_consensus.cc:2258] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 02:21:49.253166 6119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:21:49.253157 6657 raft_consensus.cc:2229] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:21:49.253526 6657 raft_consensus.cc:2258] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:49.254226 6659 ts_tablet_manager.cc:1905] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:21:49.254918 6658 ts_tablet_manager.cc:1905] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:21:49.258764 6261 tablet_service.cc:1467] Processing CreateTablet for tablet 33ec722af90d41c8b95cfa47709e07f3 (DEFAULT_TABLE table=test-table [id=c820a7c74927431e913da64f3d0cc28b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:49.259053 6261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ec722af90d41c8b95cfa47709e07f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:49.259250 6404 tablet_service.cc:1467] Processing CreateTablet for tablet 33ec722af90d41c8b95cfa47709e07f3 (DEFAULT_TABLE table=test-table [id=c820a7c74927431e913da64f3d0cc28b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:49.259507 6404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ec722af90d41c8b95cfa47709e07f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:49.259624 6658 ts_tablet_manager.cc:1918] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:21:49.259714 6658 log.cc:1196] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/wal/wals/0ddca34d4d8c42e3a60a0e9ac9200366
I20260807 02:21:49.260048 6658 ts_tablet_manager.cc:1939] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1b79e3d055c341d0be5bc3d10ebaf235: Deleting consensus metadata
I20260807 02:21:49.261348 6636 tablet_bootstrap.cc:492] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235: Bootstrap starting.
I20260807 02:21:49.262063 6545 tablet_service.cc:1467] Processing CreateTablet for tablet 33ec722af90d41c8b95cfa47709e07f3 (DEFAULT_TABLE table=test-table [id=c820a7c74927431e913da64f3d0cc28b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:49.262333 6545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ec722af90d41c8b95cfa47709e07f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:49.262356 6636 tablet_bootstrap.cc:654] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:49.263974 6108 catalog_manager.cc:4841] TS 1b79e3d055c341d0be5bc3d10ebaf235 (127.4.221.65:39137): tablet 0ddca34d4d8c42e3a60a0e9ac9200366 (table test-table [id=283d4fe82c9940cfa078d86f829443dc]) successfully deleted
I20260807 02:21:49.264410 6634 tablet_bootstrap.cc:492] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05: Bootstrap starting.
I20260807 02:21:49.264750 6636 tablet_bootstrap.cc:492] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235: No bootstrap required, opened a new log
I20260807 02:21:49.264801 6637 tablet_bootstrap.cc:492] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b: Bootstrap starting.
I20260807 02:21:49.264842 6636 ts_tablet_manager.cc:1397] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260807 02:21:49.265560 6634 tablet_bootstrap.cc:654] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:49.265743 6637 tablet_bootstrap.cc:654] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:49.265765 6636 raft_consensus.cc:348] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.266168 6636 raft_consensus.cc:374] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:49.266213 6636 raft_consensus.cc:729] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b79e3d055c341d0be5bc3d10ebaf235, State: Initialized, Role: FOLLOWER
I20260807 02:21:49.266345 6636 consensus_queue.cc:260] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.266639 6636 ts_tablet_manager.cc:1428] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260807 02:21:49.267535 6634 tablet_bootstrap.cc:492] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05: No bootstrap required, opened a new log
I20260807 02:21:49.267624 6634 ts_tablet_manager.cc:1397] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 02:21:49.268105 6634 raft_consensus.cc:348] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.268225 6634 raft_consensus.cc:374] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:49.268262 6634 raft_consensus.cc:729] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab59712909c4b988ae8b196578c3c05, State: Initialized, Role: FOLLOWER
I20260807 02:21:49.268451 6634 consensus_queue.cc:260] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.268559 6657 ts_tablet_manager.cc:1905] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:21:49.270313 6637 tablet_bootstrap.cc:492] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b: No bootstrap required, opened a new log
I20260807 02:21:49.270421 6637 ts_tablet_manager.cc:1397] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260807 02:21:49.270903 6637 raft_consensus.cc:348] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.271026 6637 raft_consensus.cc:374] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:49.271057 6637 raft_consensus.cc:729] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dfff0061e864417abc538ed2df2930b, State: Initialized, Role: FOLLOWER
I20260807 02:21:49.271178 6637 consensus_queue.cc:260] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.271459 6657 ts_tablet_manager.cc:1918] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:21:49.271483 6637 ts_tablet_manager.cc:1428] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260807 02:21:49.271584 6657 log.cc:1196] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/0ddca34d4d8c42e3a60a0e9ac9200366
I20260807 02:21:49.271880 6657 ts_tablet_manager.cc:1939] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 2ab59712909c4b988ae8b196578c3c05: Deleting consensus metadata
I20260807 02:21:49.273522 6106 catalog_manager.cc:4841] TS 2ab59712909c4b988ae8b196578c3c05 (127.4.221.66:36427): tablet 0ddca34d4d8c42e3a60a0e9ac9200366 (table test-table [id=283d4fe82c9940cfa078d86f829443dc]) successfully deleted
I20260807 02:21:49.273979 6659 ts_tablet_manager.cc:1918] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:21:49.274109 6659 log.cc:1196] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/0ddca34d4d8c42e3a60a0e9ac9200366
I20260807 02:21:49.274153 6634 ts_tablet_manager.cc:1428] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260807 02:21:49.274472 6659 ts_tablet_manager.cc:1939] T 0ddca34d4d8c42e3a60a0e9ac9200366 P 1dfff0061e864417abc538ed2df2930b: Deleting consensus metadata
I20260807 02:21:49.275619 6106 catalog_manager.cc:4841] TS 1dfff0061e864417abc538ed2df2930b (127.4.221.67:44527): tablet 0ddca34d4d8c42e3a60a0e9ac9200366 (table test-table [id=283d4fe82c9940cfa078d86f829443dc]) successfully deleted
I20260807 02:21:49.302301 6642 raft_consensus.cc:482] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:21:49.302484 6642 raft_consensus.cc:504] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.303758 6642 leader_election.cc:288] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ab59712909c4b988ae8b196578c3c05 (127.4.221.66:36427), 1dfff0061e864417abc538ed2df2930b (127.4.221.67:44527)
W20260807 02:21:49.305627 6612 tablet.cc:2369] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:21:49.309341 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.304163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:36427 (local address 127.4.221.65:55511)
0807 02:21:49.304939 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:49.304955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:49.305056 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:49.305435 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:49.305438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:49.306210 (+ 772us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:49.306346 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.306353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.307387 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.307390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.308164 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.308173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.308866 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.308870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.308980 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.309003 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 02:21:49.309082 (+ 79us) client_negotiation.cc:240] Negotiation successful
0807 02:21:49.309170 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":82,"threads_started":1}
I20260807 02:21:49.309347 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.305116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:55511 (local address 127.4.221.66:36427)
0807 02:21:49.305232 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.305235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.305247 (+ 12us) server_negotiation.cc:407] Connection header received
0807 02:21:49.305293 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.305296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.305332 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.305422 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:49.306492 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.307258 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.308312 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.308717 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.308861 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.308918 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.309133 (+ 215us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.309222 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260807 02:21:49.309423 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.304845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:45233 (local address 127.4.221.67:44527)
0807 02:21:49.304969 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.304977 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.304995 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:21:49.305126 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.305130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.305191 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.305236 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:49.305637 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.306402 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.307509 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.308166 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.308290 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.308345 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.309210 (+ 865us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.309286 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 02:21:49.309726 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.304481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:44527 (local address 127.4.221.65:45233)
0807 02:21:49.304876 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:49.304892 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:49.304988 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:49.305378 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:49.305381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:49.305396 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:49.305513 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.305518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.306499 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.306502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.307381 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.307390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.308974 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.308977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.309075 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.309096 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 02:21:49.309553 (+ 457us) client_negotiation.cc:240] Negotiation successful
0807 02:21:49.309627 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":105,"threads_started":1}
I20260807 02:21:49.309948 6424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ec722af90d41c8b95cfa47709e07f3" candidate_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab59712909c4b988ae8b196578c3c05" is_pre_election: true
I20260807 02:21:49.310098 6424 raft_consensus.cc:2454] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b79e3d055c341d0be5bc3d10ebaf235 in term 0.
I20260807 02:21:49.310487 6213 leader_election.cc:302] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b79e3d055c341d0be5bc3d10ebaf235, 2ab59712909c4b988ae8b196578c3c05; no voters:
I20260807 02:21:49.310545 6565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ec722af90d41c8b95cfa47709e07f3" candidate_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dfff0061e864417abc538ed2df2930b" is_pre_election: true
I20260807 02:21:49.310812 6642 raft_consensus.cc:2781] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:21:49.310827 6565 raft_consensus.cc:2454] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b79e3d055c341d0be5bc3d10ebaf235 in term 0.
I20260807 02:21:49.310894 6642 raft_consensus.cc:482] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:21:49.310936 6642 raft_consensus.cc:3037] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:49.312117 6642 raft_consensus.cc:504] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.312529 6642 leader_election.cc:288] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [CANDIDATE]: Term 1 election: Requested vote from peers 2ab59712909c4b988ae8b196578c3c05 (127.4.221.66:36427), 1dfff0061e864417abc538ed2df2930b (127.4.221.67:44527)
I20260807 02:21:49.312958 6424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ec722af90d41c8b95cfa47709e07f3" candidate_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab59712909c4b988ae8b196578c3c05"
I20260807 02:21:49.313009 6565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ec722af90d41c8b95cfa47709e07f3" candidate_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dfff0061e864417abc538ed2df2930b"
I20260807 02:21:49.313103 6424 raft_consensus.cc:3037] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:49.313118 6565 raft_consensus.cc:3037] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Advancing to term 1
W20260807 02:21:49.314399 6472 tablet.cc:2369] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:21:49.314438 6565 raft_consensus.cc:2454] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b79e3d055c341d0be5bc3d10ebaf235 in term 1.
I20260807 02:21:49.314438 6424 raft_consensus.cc:2454] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b79e3d055c341d0be5bc3d10ebaf235 in term 1.
I20260807 02:21:49.314884 6213 leader_election.cc:302] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b79e3d055c341d0be5bc3d10ebaf235, 2ab59712909c4b988ae8b196578c3c05; no voters:
I20260807 02:21:49.315070 6642 raft_consensus.cc:2781] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:21:49.315315 6642 raft_consensus.cc:686] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 1 LEADER]: Becoming Leader. State: Replica: 1b79e3d055c341d0be5bc3d10ebaf235, State: Running, Role: LEADER
I20260807 02:21:49.315647 6642 consensus_queue.cc:237] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.318410 6122 catalog_manager.cc:5495] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b79e3d055c341d0be5bc3d10ebaf235 (127.4.221.65). New cstate: current_term: 1 leader_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } health_report { overall_health: UNKNOWN } } }
W20260807 02:21:49.334864 6330 tablet.cc:2369] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 07 02:21:49 dist-test-slave-69rg krb5kdc[6075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069308, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:49.365738 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.355826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46163 (local address 127.0.0.1:44572)
0807 02:21:49.356168 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:49.356185 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:49.356315 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:49.356617 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:49.356619 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:49.357029 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:49.357143 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.357150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.358384 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.358386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.359022 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.359032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.359622 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.359629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.359713 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.360397 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:49.360410 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:49.362002 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:49.364086 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:49.364095 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:49.364100 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:49.364368 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:49.364686 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:49.364688 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:49.364689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:49.364775 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:49.365249 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:49.365254 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:49.365367 (+ 113us) client_negotiation.cc:769] Sending connection context
0807 02:21:49.365461 (+ 94us) client_negotiation.cc:240] Negotiation successful
0807 02:21:49.365589 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":121,"threads_started":1}
I20260807 02:21:49.365918 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.355962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.4.221.126:46163)
0807 02:21:49.356132 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.356138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.356281 (+ 143us) server_negotiation.cc:407] Connection header received
0807 02:21:49.356415 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.356419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.356475 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.356580 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:49.357282 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.358175 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.359145 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.359480 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.359597 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.362174 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:49.362180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:49.362183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:49.362216 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:49.363902 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:49.364478 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:49.364482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:49.364484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:49.364580 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:49.364874 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:49.364878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:49.364879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:49.365133 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:49.365242 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.365556 (+ 314us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.365690 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 02:21:49.370718 6122 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:44572:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 02:21:49.371670 6122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:21:49.379199 6404 tablet_service.cc:1467] Processing CreateTablet for tablet 9d448f755a4a469aae7264e7a8959d00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34076de788b54c17b83ddaf8806c17ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:21:49.379637 6404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d448f755a4a469aae7264e7a8959d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:49.379149 6261 tablet_service.cc:1467] Processing CreateTablet for tablet 9d448f755a4a469aae7264e7a8959d00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34076de788b54c17b83ddaf8806c17ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:21:49.382210 6261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d448f755a4a469aae7264e7a8959d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:49.382289 6545 tablet_service.cc:1467] Processing CreateTablet for tablet 9d448f755a4a469aae7264e7a8959d00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=34076de788b54c17b83ddaf8806c17ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:21:49.382658 6545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d448f755a4a469aae7264e7a8959d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:49.382676 6634 tablet_bootstrap.cc:492] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05: Bootstrap starting.
I20260807 02:21:49.383765 6634 tablet_bootstrap.cc:654] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:49.385218 6637 tablet_bootstrap.cc:492] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b: Bootstrap starting.
I20260807 02:21:49.386216 6634 tablet_bootstrap.cc:492] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05: No bootstrap required, opened a new log
I20260807 02:21:49.386270 6637 tablet_bootstrap.cc:654] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:49.386319 6634 ts_tablet_manager.cc:1397] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 02:21:49.386817 6634 raft_consensus.cc:348] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.386961 6634 raft_consensus.cc:374] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:49.387008 6634 raft_consensus.cc:729] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ab59712909c4b988ae8b196578c3c05, State: Initialized, Role: FOLLOWER
I20260807 02:21:49.387302 6634 consensus_queue.cc:260] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.387645 6637 tablet_bootstrap.cc:492] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b: No bootstrap required, opened a new log
I20260807 02:21:49.387732 6637 ts_tablet_manager.cc:1397] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260807 02:21:49.387646 6641 tablet_replica.cc:439] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } }
I20260807 02:21:49.387814 6641 tablet_replica.cc:442] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:21:49.388185 6637 raft_consensus.cc:348] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.388300 6634 ts_tablet_manager.cc:1428] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260807 02:21:49.388314 6637 raft_consensus.cc:374] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:49.388350 6637 raft_consensus.cc:729] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dfff0061e864417abc538ed2df2930b, State: Initialized, Role: FOLLOWER
I20260807 02:21:49.388477 6637 consensus_queue.cc:260] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.388705 6643 tablet_replica.cc:439] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } }
I20260807 02:21:49.388904 6643 tablet_replica.cc:442] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:21:49.389088 6637 ts_tablet_manager.cc:1428] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 02:21:49.393040 6636 tablet_bootstrap.cc:492] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235: Bootstrap starting.
I20260807 02:21:49.394145 6636 tablet_bootstrap.cc:654] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:49.395385 6636 tablet_bootstrap.cc:492] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235: No bootstrap required, opened a new log
I20260807 02:21:49.395473 6636 ts_tablet_manager.cc:1397] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 02:21:49.395946 6636 raft_consensus.cc:348] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.396100 6636 raft_consensus.cc:374] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:49.396128 6636 raft_consensus.cc:729] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b79e3d055c341d0be5bc3d10ebaf235, State: Initialized, Role: FOLLOWER
I20260807 02:21:49.396282 6636 consensus_queue.cc:260] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.396566 6636 ts_tablet_manager.cc:1428] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 02:21:49.396520 6642 tablet_replica.cc:439] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } }
I20260807 02:21:49.396652 6642 tablet_replica.cc:442] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:21:49.521351 6643 raft_consensus.cc:482] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:21:49.521514 6643 raft_consensus.cc:504] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.522107 6643 leader_election.cc:288] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b79e3d055c341d0be5bc3d10ebaf235 (127.4.221.65:39137), 2ab59712909c4b988ae8b196578c3c05 (127.4.221.66:36427)
I20260807 02:21:49.522646 6424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d448f755a4a469aae7264e7a8959d00" candidate_uuid: "1dfff0061e864417abc538ed2df2930b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab59712909c4b988ae8b196578c3c05" is_pre_election: true
I20260807 02:21:49.522655 6281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d448f755a4a469aae7264e7a8959d00" candidate_uuid: "1dfff0061e864417abc538ed2df2930b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" is_pre_election: true
I20260807 02:21:49.522837 6424 raft_consensus.cc:2454] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dfff0061e864417abc538ed2df2930b in term 0.
I20260807 02:21:49.522842 6281 raft_consensus.cc:2454] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dfff0061e864417abc538ed2df2930b in term 0.
I20260807 02:21:49.523197 6499 leader_election.cc:302] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b79e3d055c341d0be5bc3d10ebaf235, 1dfff0061e864417abc538ed2df2930b; no voters:
I20260807 02:21:49.523404 6643 raft_consensus.cc:2781] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:21:49.523463 6643 raft_consensus.cc:482] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:21:49.523504 6643 raft_consensus.cc:3037] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:49.524540 6643 raft_consensus.cc:504] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.524924 6643 leader_election.cc:288] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [CANDIDATE]: Term 1 election: Requested vote from peers 1b79e3d055c341d0be5bc3d10ebaf235 (127.4.221.65:39137), 2ab59712909c4b988ae8b196578c3c05 (127.4.221.66:36427)
I20260807 02:21:49.525283 6281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d448f755a4a469aae7264e7a8959d00" candidate_uuid: "1dfff0061e864417abc538ed2df2930b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b79e3d055c341d0be5bc3d10ebaf235"
I20260807 02:21:49.525283 6424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d448f755a4a469aae7264e7a8959d00" candidate_uuid: "1dfff0061e864417abc538ed2df2930b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ab59712909c4b988ae8b196578c3c05"
I20260807 02:21:49.525434 6281 raft_consensus.cc:3037] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:49.525457 6424 raft_consensus.cc:3037] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:49.526619 6281 raft_consensus.cc:2454] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dfff0061e864417abc538ed2df2930b in term 1.
I20260807 02:21:49.526619 6424 raft_consensus.cc:2454] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dfff0061e864417abc538ed2df2930b in term 1.
I20260807 02:21:49.527005 6499 leader_election.cc:302] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b79e3d055c341d0be5bc3d10ebaf235, 1dfff0061e864417abc538ed2df2930b; no voters:
I20260807 02:21:49.527191 6643 raft_consensus.cc:2781] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:21:49.527305 6643 raft_consensus.cc:686] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [term 1 LEADER]: Becoming Leader. State: Replica: 1dfff0061e864417abc538ed2df2930b, State: Running, Role: LEADER
I20260807 02:21:49.527446 6643 consensus_queue.cc:237] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } }
I20260807 02:21:49.528129 6645 tablet_replica.cc:439] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b: TxnStatusTablet state changed. Reason: New leader 1dfff0061e864417abc538ed2df2930b. Latest consensus state: current_term: 1 leader_uuid: "1dfff0061e864417abc538ed2df2930b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } }
I20260807 02:21:49.528236 6645 tablet_replica.cc:442] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:21:49.528489 6678 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 02:21:49.528687 6678 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 02:21:49.529587 6120 catalog_manager.cc:5495] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dfff0061e864417abc538ed2df2930b (127.4.221.67). New cstate: current_term: 1 leader_uuid: "1dfff0061e864417abc538ed2df2930b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } health_report { overall_health: HEALTHY } } }
I20260807 02:21:49.547513 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.543996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:44527 (local address 127.0.0.1:49560)
0807 02:21:49.544198 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:49.544214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:49.544344 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:49.544656 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:49.544659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:49.544667 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:21:49.544793 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.544798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.545544 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.545548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.546249 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.546259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.546722 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.546726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.546811 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.546859 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:21:49.547182 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:21:49.547201 (+ 19us) client_negotiation.cc:769] Sending connection context
0807 02:21:49.547286 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 02:21:49.547370 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260807 02:21:49.547513 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.544109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49560 (local address 127.4.221.67:44527)
0807 02:21:49.544251 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.544258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.544274 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:21:49.544477 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.544480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.544538 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.544620 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:21:49.544905 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.545429 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.546371 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.546596 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.546712 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.547005 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:21:49.547090 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:21:49.547174 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.547300 (+ 126us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.547358 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 02:21:49.550899 6424 raft_consensus.cc:1261] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05 [term 1 FOLLOWER]: Refusing update from remote peer 1dfff0061e864417abc538ed2df2930b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:49.551049 6281 raft_consensus.cc:1261] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 1 FOLLOWER]: Refusing update from remote peer 1dfff0061e864417abc538ed2df2930b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:49.551402 6645 consensus_queue.cc:1035] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:21:49.551560 6643 consensus_queue.cc:1035] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:21:49.553125 6641 tablet_replica.cc:439] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05: TxnStatusTablet state changed. Reason: New leader 1dfff0061e864417abc538ed2df2930b. Latest consensus state: current_term: 1 leader_uuid: "1dfff0061e864417abc538ed2df2930b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } }
I20260807 02:21:49.553289 6641 tablet_replica.cc:442] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:21:49.554402 6642 tablet_replica.cc:439] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235: TxnStatusTablet state changed. Reason: New leader 1dfff0061e864417abc538ed2df2930b. Latest consensus state: current_term: 1 leader_uuid: "1dfff0061e864417abc538ed2df2930b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } }
I20260807 02:21:49.554533 6642 tablet_replica.cc:442] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:21:49.556298 6643 tablet_replica.cc:439] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1dfff0061e864417abc538ed2df2930b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } }
I20260807 02:21:49.556424 6643 tablet_replica.cc:442] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:21:49.556550 6643 tablet_replica.cc:439] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1dfff0061e864417abc538ed2df2930b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } }
I20260807 02:21:49.556620 6643 tablet_replica.cc:442] T 9d448f755a4a469aae7264e7a8959d00 P 1dfff0061e864417abc538ed2df2930b: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:21:49.557345 6642 tablet_replica.cc:439] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1dfff0061e864417abc538ed2df2930b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } }
I20260807 02:21:49.557456 6642 tablet_replica.cc:442] T 9d448f755a4a469aae7264e7a8959d00 P 1b79e3d055c341d0be5bc3d10ebaf235: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:21:49.559134 6641 tablet_replica.cc:439] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1dfff0061e864417abc538ed2df2930b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b79e3d055c341d0be5bc3d10ebaf235" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39137 } } peers { permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 } } peers { permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 } } }
I20260807 02:21:49.559283 6641 tablet_replica.cc:442] T 9d448f755a4a469aae7264e7a8959d00 P 2ab59712909c4b988ae8b196578c3c05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:21:49.559283 6649 mvcc.cc:205] Tried to move back new op lower bound from 7315739891915755520 to 7315739891825946624. Current Snapshot: MvccSnapshot[applied={T|T < 7315739891915755520 or (T in {7315739891915755520})}]
I20260807 02:21:49.566931 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.562822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.4.221.65:39137)
0807 02:21:49.562959 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.562963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.562977 (+ 14us) server_negotiation.cc:407] Connection header received
0807 02:21:49.563102 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.563104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.563160 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.563278 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:21:49.563540 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.564047 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.565244 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.565545 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.565662 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.565971 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:21:49.566478 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:21:49.566571 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.566675 (+ 104us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.566731 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 02:21:49.579021 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.575113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:56583 (local address 127.4.221.67:44527)
0807 02:21:49.575268 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.575273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.575460 (+ 187us) server_negotiation.cc:407] Connection header received
0807 02:21:49.575578 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.575580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.575639 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.575719 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:21:49.576045 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.576577 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.577911 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.578159 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.578252 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.578592 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:21:49.578678 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:21:49.578736 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.578856 (+ 120us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.578907 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 02:21:49.579468 6686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.575011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:44527 (local address 127.4.221.65:56583)
0807 02:21:49.575368 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:49.575383 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:49.575494 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:49.575746 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:49.575749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:49.575757 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:21:49.575905 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.575911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.576706 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.576709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.577715 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.577726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.578278 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.578282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.578391 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.578450 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:21:49.578761 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:21:49.578768 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 02:21:49.578837 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 02:21:49.578922 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":138,"threads_started":1}
I20260807 02:21:49.589095 6565 raft_consensus.cc:1261] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b [term 1 FOLLOWER]: Refusing update from remote peer 1b79e3d055c341d0be5bc3d10ebaf235: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:49.589108 6424 raft_consensus.cc:1261] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05 [term 1 FOLLOWER]: Refusing update from remote peer 1b79e3d055c341d0be5bc3d10ebaf235: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:49.589785 6666 consensus_queue.cc:1035] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ab59712909c4b988ae8b196578c3c05" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:21:49.590245 6642 consensus_queue.cc:1035] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dfff0061e864417abc538ed2df2930b" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:21:49.618913 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.615124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:37827 (local address 127.4.221.65:39137)
0807 02:21:49.615247 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.615251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.615266 (+ 15us) server_negotiation.cc:407] Connection header received
0807 02:21:49.615320 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.615322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.615367 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.615468 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:21:49.615700 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.616200 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.617483 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.617929 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.618005 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.618428 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:21:49.618512 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:21:49.618573 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.618687 (+ 114us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.618768 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260807 02:21:49.618986 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.614690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:39137 (local address 127.4.221.67:37827)
0807 02:21:49.615055 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:49.615072 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:49.615210 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:49.615480 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:49.615482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:49.615490 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:21:49.615605 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.615612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.616362 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.616366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.617337 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:49.617348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.618133 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.618137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:49.618228 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.618283 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:21:49.618602 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:21:49.618609 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 02:21:49.618680 (+ 71us) client_negotiation.cc:240] Negotiation successful
0807 02:21:49.618775 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":131,"threads_started":1}
I20260807 02:21:49.650012 6120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:21:49.650224 6120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:21:49.652896 6120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcfc9dea13804b999d155e0782244a98: Sending DeleteTablet for 3 replicas of tablet 33ec722af90d41c8b95cfa47709e07f3
I20260807 02:21:49.653872 6261 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ec722af90d41c8b95cfa47709e07f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:49 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:47030
I20260807 02:21:49.654074 6658 tablet_replica.cc:330] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235: stopping tablet replica
I20260807 02:21:49.654129 6404 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ec722af90d41c8b95cfa47709e07f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:49 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:44316
I20260807 02:21:49.654208 6658 raft_consensus.cc:2229] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:21:49.654377 6657 tablet_replica.cc:330] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05: stopping tablet replica
I20260807 02:21:49.654500 6657 raft_consensus.cc:2229] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:21:49.654580 6658 raft_consensus.cc:2258] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:49.654613 6657 raft_consensus.cc:2258] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:49.655083 6545 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ec722af90d41c8b95cfa47709e07f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:49 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:36806
I20260807 02:21:49.655300 6659 tablet_replica.cc:330] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b: stopping tablet replica
I20260807 02:21:49.655423 6659 raft_consensus.cc:2229] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:21:49.655481 6658 ts_tablet_manager.cc:1905] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:21:49.655547 6659 raft_consensus.cc:2258] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:49.659303 6659 ts_tablet_manager.cc:1905] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:21:49.662623 6657 ts_tablet_manager.cc:1905] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 07 02:21:49 dist-test-slave-69rg krb5kdc[6075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069309, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.221.65@KRBTEST.COM
I20260807 02:21:49.665571 6657 ts_tablet_manager.cc:1918] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:21:49.665683 6657 log.cc:1196] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/33ec722af90d41c8b95cfa47709e07f3
I20260807 02:21:49.666437 6659 ts_tablet_manager.cc:1918] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:21:49.666529 6659 log.cc:1196] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/33ec722af90d41c8b95cfa47709e07f3
I20260807 02:21:49.666826 6659 ts_tablet_manager.cc:1939] T 33ec722af90d41c8b95cfa47709e07f3 P 1dfff0061e864417abc538ed2df2930b: Deleting consensus metadata
I20260807 02:21:49.671573 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.657104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32936 (local address 127.4.221.65:39137)
0807 02:21:49.657239 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.657246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.657853 (+ 607us) server_negotiation.cc:407] Connection header received
0807 02:21:49.658087 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.658091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.658176 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.658232 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:49.659262 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.660046 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.661390 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.661751 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.661862 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.666460 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:49.666477 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:49.666496 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:49.666532 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:49.669092 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:49.669748 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:49.669756 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:49.669758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:49.669889 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:49.670295 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:49.670299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:49.670301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:49.670631 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:49.670812 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.671246 (+ 434us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.671387 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 02:21:49.671746 6658 ts_tablet_manager.cc:1918] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:21:49.671834 6658 log.cc:1196] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786069301823597-4981-0/minicluster-data/ts-0/wal/wals/33ec722af90d41c8b95cfa47709e07f3
I20260807 02:21:49.672128 6658 ts_tablet_manager.cc:1939] T 33ec722af90d41c8b95cfa47709e07f3 P 1b79e3d055c341d0be5bc3d10ebaf235: Deleting consensus metadata
I20260807 02:21:49.672469 6657 ts_tablet_manager.cc:1939] T 33ec722af90d41c8b95cfa47709e07f3 P 2ab59712909c4b988ae8b196578c3c05: Deleting consensus metadata
W20260807 02:21:49.673214 6323 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32936
I20260807 02:21:49.673676 6106 catalog_manager.cc:4841] TS 1dfff0061e864417abc538ed2df2930b (127.4.221.67:44527): tablet 33ec722af90d41c8b95cfa47709e07f3 (table test-table [id=c820a7c74927431e913da64f3d0cc28b]) successfully deleted
I20260807 02:21:49.674363 6108 catalog_manager.cc:4841] TS 1b79e3d055c341d0be5bc3d10ebaf235 (127.4.221.65:39137): tablet 33ec722af90d41c8b95cfa47709e07f3 (table test-table [id=c820a7c74927431e913da64f3d0cc28b]) successfully deleted
I20260807 02:21:49.674373 6106 catalog_manager.cc:4841] TS 2ab59712909c4b988ae8b196578c3c05 (127.4.221.66:36427): tablet 33ec722af90d41c8b95cfa47709e07f3 (table test-table [id=c820a7c74927431e913da64f3d0cc28b]) successfully deleted
I20260807 02:21:49.685561 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:49.675875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.4.221.126:46163)
0807 02:21:49.676003 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:49.676008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:49.676481 (+ 473us) server_negotiation.cc:407] Connection header received
0807 02:21:49.676583 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:49.676586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:49.676639 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:49.676727 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:49.677545 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.678473 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.679576 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:49.679922 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:49.680030 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:49.681774 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:49.681778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:49.681781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:49.681812 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:49.683587 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:49.684078 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:49.684081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:49.684083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:49.684184 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:49.684573 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:49.684576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:49.684578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:49.684820 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:49.684955 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:49.685243 (+ 288us) server_negotiation.cc:299] Negotiation successful
0807 02:21:49.685384 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260807 02:21:49.686335 6120 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44574
I20260807 02:21:49.687938 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 6195
I20260807 02:21:49.695807 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 6338
I20260807 02:21:49.703435 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 6480
I20260807 02:21:49.711362 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 6088
2026-08-07T02:21:49Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2839 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:21:49 dist-test-slave-69rg krb5kdc[6701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:21:49 dist-test-slave-69rg krb5kdc[6701](info): set up 2 sockets
Aug 07 02:21:49 dist-test-slave-69rg krb5kdc[6701](info): commencing operation
krb5kdc: starting...
W20260807 02:21:50.797010 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:21:50 dist-test-slave-69rg krb5kdc[6701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:21:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:21:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:21:50.937798 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:46169
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:46169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:51.054996 6714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:51.055375 6714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:51.055455 6714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:51.059928 6714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:21:51.060043 6714 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 02:21:51.060073 6714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:51.060119 6714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:51.060178 6714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:21:51.060220 6714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:21:51.066077 6714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:46169
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:46169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:51.067344 6714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:51.068457 6714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:51.077585 6720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:51.077567 6722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:51.077661 6714 server_base.cc:1034] running on GCE node
W20260807 02:21:51.077569 6719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:51.078544 6714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:51.080089 6714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:51.081364 6714 hybrid_clock.cc:648] HybridClock initialized: now 1786069311081349 us; error 34 us; skew 500 ppm
Aug 07 02:21:51 dist-test-slave-69rg krb5kdc[6701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069311, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:51.088708 6714 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:21:51.089895 6714 webserver.cc:460] Webserver started at http://127.4.221.126:33317/ using document root <none> and password file <none>
I20260807 02:21:51.090523 6714 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:51.090590 6714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:51.090778 6714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:51.092931 6714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "d66968a8cb1c4e70aadc535b7a9e33b5"
format_stamp: "Formatted at 2026-08-07 02:21:51 on dist-test-slave-69rg"
server_key: "0e59b8485e309b5321a59b30ccdcabd6"
server_key_iv: "a5042854b0e5cf54f05fda1eae5e1077"
server_key_version: "encryptionkey@0"
I20260807 02:21:51.093487 6714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "d66968a8cb1c4e70aadc535b7a9e33b5"
format_stamp: "Formatted at 2026-08-07 02:21:51 on dist-test-slave-69rg"
server_key: "0e59b8485e309b5321a59b30ccdcabd6"
server_key_iv: "a5042854b0e5cf54f05fda1eae5e1077"
server_key_version: "encryptionkey@0"
I20260807 02:21:51.097625 6714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260807 02:21:51.100478 6729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:51.102023 6714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:21:51.102146 6714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "d66968a8cb1c4e70aadc535b7a9e33b5"
format_stamp: "Formatted at 2026-08-07 02:21:51 on dist-test-slave-69rg"
server_key: "0e59b8485e309b5321a59b30ccdcabd6"
server_key_iv: "a5042854b0e5cf54f05fda1eae5e1077"
server_key_version: "encryptionkey@0"
I20260807 02:21:51.102258 6714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:51.117100 6714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:51.120234 6714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:51.120410 6714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:51.129340 6714 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:46169
I20260807 02:21:51.129359 6792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:46169 every 8 connection(s)
I20260807 02:21:51.130440 6714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:21:51.133735 6793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:51.136144 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 6714
I20260807 02:21:51.136281 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:21:51.136592 4981 external_mini_cluster.cc:1428] Setting key 24739262741ab1790b8fb11ae6f681fc
I20260807 02:21:51.141506 6793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5: Bootstrap starting.
I20260807 02:21:51.144368 6793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:51.145438 6793 log.cc:824] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5: Log is configured to *not* fsync() on all Append() calls
Aug 07 02:21:51 dist-test-slave-69rg krb5kdc[6701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:51.148303 6793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5: No bootstrap required, opened a new log
I20260807 02:21:51.152068 6793 raft_consensus.cc:348] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66968a8cb1c4e70aadc535b7a9e33b5" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46169 } }
I20260807 02:21:51.152400 6793 raft_consensus.cc:374] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:51.152460 6793 raft_consensus.cc:729] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66968a8cb1c4e70aadc535b7a9e33b5, State: Initialized, Role: FOLLOWER
I20260807 02:21:51.153026 6793 consensus_queue.cc:260] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66968a8cb1c4e70aadc535b7a9e33b5" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46169 } }
I20260807 02:21:51.153228 6793 raft_consensus.cc:388] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:21:51.153329 6793 raft_consensus.cc:482] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:21:51.153481 6793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:51.154235 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.138289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.4.221.126:46169)
0807 02:21:51.138784 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:51.138804 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:51.138841 (+ 37us) server_negotiation.cc:407] Connection header received
0807 02:21:51.139585 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:51.139604 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:51.139952 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:51.140322 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:51.141222 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.142441 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.143370 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.143906 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.144098 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.147861 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:51.147868 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:51.147886 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:51.147925 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:51.150222 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.150818 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.150825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.150831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.150966 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.151321 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.151327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.151329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.151824 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:51.152104 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:51.152465 (+ 361us) server_negotiation.cc:299] Negotiation successful
0807 02:21:51.152903 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":159,"threads_started":1}
I20260807 02:21:51.154850 6793 raft_consensus.cc:504] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66968a8cb1c4e70aadc535b7a9e33b5" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46169 } }
I20260807 02:21:51.155217 6793 leader_election.cc:302] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d66968a8cb1c4e70aadc535b7a9e33b5; no voters:
I20260807 02:21:51.155604 6793 leader_election.cc:288] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:21:51.155746 6799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:21:51.156019 6799 raft_consensus.cc:686] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [term 1 LEADER]: Becoming Leader. State: Replica: d66968a8cb1c4e70aadc535b7a9e33b5, State: Running, Role: LEADER
I20260807 02:21:51.156365 6799 consensus_queue.cc:237] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66968a8cb1c4e70aadc535b7a9e33b5" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46169 } }
I20260807 02:21:51.156951 6793 sys_catalog.cc:564] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:21:51.157675 6800 sys_catalog.cc:455] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d66968a8cb1c4e70aadc535b7a9e33b5. Latest consensus state: current_term: 1 leader_uuid: "d66968a8cb1c4e70aadc535b7a9e33b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66968a8cb1c4e70aadc535b7a9e33b5" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46169 } } }
I20260807 02:21:51.157859 6800 sys_catalog.cc:458] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [sys.catalog]: This master's current role is: LEADER
I20260807 02:21:51.158272 6801 sys_catalog.cc:455] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d66968a8cb1c4e70aadc535b7a9e33b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d66968a8cb1c4e70aadc535b7a9e33b5" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46169 } } }
I20260807 02:21:51.158387 6801 sys_catalog.cc:458] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5 [sys.catalog]: This master's current role is: LEADER
W20260807 02:21:51.162271 6814 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:21:51.162381 6814 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:21:51.162496 6808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:21:51.163302 6808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:21:51.169552 6808 catalog_manager.cc:1347] Generated new cluster ID: d714db1388d74a54acb149961b933704
I20260807 02:21:51.169656 6808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:21:51.183442 6808 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:21:51.184605 6808 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:21:51.193572 6808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5: Generated new TSK 0
I20260807 02:21:51.194525 6808 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:21:51.233570 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:46169
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:51.354279 6820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:51.354626 6820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:51.354713 6820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:51.359203 6820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:51.359299 6820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:51.359371 6820 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 02:21:51.359403 6820 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:21:51.359472 6820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:21:51.365195 6820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.221.126:46169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:51.366551 6820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:51.367672 6820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:51.377859 6828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:51.377940 6820 server_base.cc:1034] running on GCE node
W20260807 02:21:51.377859 6825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:51.377935 6826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:51.378607 6820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:51.379280 6820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:51.380456 6820 hybrid_clock.cc:648] HybridClock initialized: now 1786069311380437 us; error 32 us; skew 500 ppm
Aug 07 02:21:51 dist-test-slave-69rg krb5kdc[6701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069311, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:51.383625 6820 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:21:51.384673 6820 webserver.cc:460] Webserver started at http://127.4.221.65:44921/ using document root <none> and password file <none>
I20260807 02:21:51.385308 6820 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:51.385408 6820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:51.385660 6820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:51.387903 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "a98433a0397243a8885c782f775af375"
format_stamp: "Formatted at 2026-08-07 02:21:51 on dist-test-slave-69rg"
server_key: "0ebd44e1908886f846c0e4231ddc31e9"
server_key_iv: "2466ed6cdd51030cc6e16268e112a59c"
server_key_version: "encryptionkey@0"
I20260807 02:21:51.388515 6820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "a98433a0397243a8885c782f775af375"
format_stamp: "Formatted at 2026-08-07 02:21:51 on dist-test-slave-69rg"
server_key: "0ebd44e1908886f846c0e4231ddc31e9"
server_key_iv: "2466ed6cdd51030cc6e16268e112a59c"
server_key_version: "encryptionkey@0"
I20260807 02:21:51.392647 6820 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:21:51.395445 6835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:51.396827 6820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:21:51.396998 6820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "a98433a0397243a8885c782f775af375"
format_stamp: "Formatted at 2026-08-07 02:21:51 on dist-test-slave-69rg"
server_key: "0ebd44e1908886f846c0e4231ddc31e9"
server_key_iv: "2466ed6cdd51030cc6e16268e112a59c"
server_key_version: "encryptionkey@0"
I20260807 02:21:51.397156 6820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:51.413911 6820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:51.417603 6820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:51.417990 6820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:51.419754 6820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:21:51.419837 6820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:51.419909 6820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:21:51.419952 6820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:21:51 dist-test-slave-69rg krb5kdc[6701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069311, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:51.431692 6820 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:35129
I20260807 02:21:51.431758 6953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:35129 every 8 connection(s)
I20260807 02:21:51.432755 6820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:21:51.436578 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.420755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:60429 (local address 127.4.221.126:46169)
0807 02:21:51.420901 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:51.420905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:51.422166 (+ 1261us) server_negotiation.cc:407] Connection header received
0807 02:21:51.423325 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:51.423328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:51.423396 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:51.423512 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:51.425413 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.426146 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.428194 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.428529 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.428624 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.432334 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:51.432338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:51.432341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:51.432381 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:51.434171 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.434825 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.434829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.434831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.434929 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.435314 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.435317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.435319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.435548 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:51.435671 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:51.436234 (+ 563us) server_negotiation.cc:299] Negotiation successful
0807 02:21:51.436387 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 02:21:51.437451 6844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.421306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46169 (local address 127.4.221.65:60429)
0807 02:21:51.421998 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:51.422048 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:51.423058 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:51.423787 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:51.423800 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:51.424344 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:51.425176 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.425203 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.426275 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.426282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.427993 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.428011 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.428732 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.428739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.428994 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.429950 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:51.429978 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:51.432079 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:51.434305 (+ 2226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:51.434313 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:51.434330 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:51.434703 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:51.435043 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:51.435046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:51.435048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:51.435209 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:51.435673 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:51.435683 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:51.435964 (+ 281us) client_negotiation.cc:769] Sending connection context
0807 02:21:51.436167 (+ 203us) client_negotiation.cc:240] Negotiation successful
0807 02:21:51.436535 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":150,"threads_started":1}
I20260807 02:21:51.439080 6954 heartbeater.cc:347] Connected to a master server at 127.4.221.126:46169
I20260807 02:21:51.439273 6954 heartbeater.cc:464] Registering TS with master...
I20260807 02:21:51.439929 6954 heartbeater.cc:511] Master 127.4.221.126:46169 requested a full tablet report, sending...
I20260807 02:21:51.441989 6746 ts_manager.cc:194] Registered new tserver with Master: a98433a0397243a8885c782f775af375 (127.4.221.65:35129)
I20260807 02:21:51.442535 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 6820
I20260807 02:21:51.442678 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:21:51.443002 4981 external_mini_cluster.cc:1428] Setting key 24976ecbbaa2acd26ceace0937f61bc3
I20260807 02:21:51.443590 6746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:60429
I20260807 02:21:51.450089 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.441394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46169 (local address 127.4.221.65:60799)
0807 02:21:51.441748 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:51.441766 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:51.441914 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:51.442220 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:51.442223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:51.442527 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:51.442696 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.442710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.443829 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.443832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.444690 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.444702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.445146 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.445153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.445283 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.445917 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:51.445938 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:51.446462 (+ 524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:51.448377 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:51.448380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:51.448383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:51.448611 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:51.448924 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:51.448927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:51.448929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:51.449027 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:51.449451 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:51.449455 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:51.449559 (+ 104us) client_negotiation.cc:769] Sending connection context
0807 02:21:51.449676 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 02:21:51.449881 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":114,"threads_started":1}
I20260807 02:21:51.450076 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.441535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:60799 (local address 127.4.221.126:46169)
0807 02:21:51.441690 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:51.441693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:51.441908 (+ 215us) server_negotiation.cc:407] Connection header received
0807 02:21:51.442006 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:51.442008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:51.442051 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:51.442169 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:51.442886 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.443645 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.444800 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.445015 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.445099 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.446618 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:51.446621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:51.446623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:51.446660 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:51.448237 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.448721 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.448725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.448727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.448815 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.449141 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.449144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.449146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.449348 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:51.449462 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:51.449739 (+ 277us) server_negotiation.cc:299] Negotiation successful
0807 02:21:51.449895 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:21:51.477137 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:46169
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:51.595459 6964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:51.595759 6964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:51.595852 6964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:51.600134 6964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:51.600207 6964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:51.600301 6964 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 02:21:51.600339 6964 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:21:51.600385 6964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:21:51.606158 6964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.221.126:46169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:51.607441 6964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:51.608489 6964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:51.618583 6972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:51.618633 6964 server_base.cc:1034] running on GCE node
W20260807 02:21:51.618577 6970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:51.618577 6969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:51.619298 6964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:51.619962 6964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:51.621127 6964 hybrid_clock.cc:648] HybridClock initialized: now 1786069311621107 us; error 34 us; skew 500 ppm
Aug 07 02:21:51 dist-test-slave-69rg krb5kdc[6701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069311, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:51.624272 6964 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:21:51.625358 6964 webserver.cc:460] Webserver started at http://127.4.221.66:34763/ using document root <none> and password file <none>
I20260807 02:21:51.626045 6964 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:51.626118 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:51.626349 6964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:51.628582 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "1dcfdf2cb750443584931b0a7ab8ae4c"
format_stamp: "Formatted at 2026-08-07 02:21:51 on dist-test-slave-69rg"
server_key: "83766d8f815eba7daf86ff4db1c1915f"
server_key_iv: "b91ba49486ef7b5c24e323186da2eb28"
server_key_version: "encryptionkey@0"
I20260807 02:21:51.629247 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "1dcfdf2cb750443584931b0a7ab8ae4c"
format_stamp: "Formatted at 2026-08-07 02:21:51 on dist-test-slave-69rg"
server_key: "83766d8f815eba7daf86ff4db1c1915f"
server_key_iv: "b91ba49486ef7b5c24e323186da2eb28"
server_key_version: "encryptionkey@0"
I20260807 02:21:51.633551 6964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260807 02:21:51.636343 6979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:51.637739 6964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:21:51.637914 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "1dcfdf2cb750443584931b0a7ab8ae4c"
format_stamp: "Formatted at 2026-08-07 02:21:51 on dist-test-slave-69rg"
server_key: "83766d8f815eba7daf86ff4db1c1915f"
server_key_iv: "b91ba49486ef7b5c24e323186da2eb28"
server_key_version: "encryptionkey@0"
I20260807 02:21:51.638060 6964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:51.658218 6964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:51.661988 6964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:51.662226 6964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:51.663895 6964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:21:51.663980 6964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:51.664062 6964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:21:51.664105 6964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:21:51 dist-test-slave-69rg krb5kdc[6701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069311, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:51.676146 6964 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:39693
I20260807 02:21:51.676164 7097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:39693 every 8 connection(s)
I20260807 02:21:51.677420 6964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:21:51.680457 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.664844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:56219 (local address 127.4.221.126:46169)
0807 02:21:51.664983 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:51.664987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:51.666323 (+ 1336us) server_negotiation.cc:407] Connection header received
0807 02:21:51.667491 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:51.667495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:51.667564 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:51.667672 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:51.669456 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.670272 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.672232 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.672604 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.672731 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.676255 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:51.676262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:51.676266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:51.676301 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:51.678596 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.679158 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.679161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.679162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.679249 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.679491 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.679492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.679493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.679668 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:51.679748 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:51.680216 (+ 468us) server_negotiation.cc:299] Negotiation successful
0807 02:21:51.680330 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 02:21:51.681318 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.665520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46169 (local address 127.4.221.66:56219)
0807 02:21:51.666135 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:51.666193 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:51.667226 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:51.667925 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:51.667934 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:51.668511 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:51.669267 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.669295 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.670428 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.670432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.672066 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.672083 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.672801 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.672806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.673009 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.673713 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:51.673749 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:51.676027 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:51.678726 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:51.678732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:51.678747 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:51.679071 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:51.679310 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:51.679312 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:51.679313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:51.679436 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:51.679757 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:51.679762 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:51.680003 (+ 241us) client_negotiation.cc:769] Sending connection context
0807 02:21:51.680164 (+ 161us) client_negotiation.cc:240] Negotiation successful
0807 02:21:51.680427 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":191,"threads_started":1}
I20260807 02:21:51.682657 7098 heartbeater.cc:347] Connected to a master server at 127.4.221.126:46169
I20260807 02:21:51.682806 7098 heartbeater.cc:464] Registering TS with master...
I20260807 02:21:51.683400 7098 heartbeater.cc:511] Master 127.4.221.126:46169 requested a full tablet report, sending...
I20260807 02:21:51.684713 6745 ts_manager.cc:194] Registered new tserver with Master: 1dcfdf2cb750443584931b0a7ab8ae4c (127.4.221.66:39693)
I20260807 02:21:51.685560 6745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:56219
I20260807 02:21:51.685923 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 6964
I20260807 02:21:51.686053 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:21:51.686406 4981 external_mini_cluster.cc:1428] Setting key a95c47a5ab74905785acd5679bebbb75
I20260807 02:21:51.693938 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.685224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:49675 (local address 127.4.221.126:46169)
0807 02:21:51.685354 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:51.685361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:51.685533 (+ 172us) server_negotiation.cc:407] Connection header received
0807 02:21:51.685650 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:51.685653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:51.685711 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:51.685803 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:51.686461 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.687134 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.688210 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.688574 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.688702 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.690094 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:51.690098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:51.690100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:51.690129 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:51.691684 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.692293 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.692296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.692297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.692384 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.692756 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.692760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.692762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.692998 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:51.693083 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:51.693551 (+ 468us) server_negotiation.cc:299] Negotiation successful
0807 02:21:51.693669 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 02:21:51.694075 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.685103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46169 (local address 127.4.221.66:49675)
0807 02:21:51.685441 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:51.685456 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:51.685556 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:51.685855 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:51.685859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:51.686200 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:51.686344 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.686350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.687270 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.687274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.688067 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.688079 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.688700 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.688703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.688778 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.689419 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:51.689433 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:51.689908 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:51.691826 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:51.691833 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:51.691839 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:51.692169 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:51.692496 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:51.692503 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:51.692508 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:51.692630 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:51.693121 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:51.693127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:51.693312 (+ 185us) client_negotiation.cc:769] Sending connection context
0807 02:21:51.693474 (+ 162us) client_negotiation.cc:240] Negotiation successful
0807 02:21:51.693697 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:21:51.721907 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:46169
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:51.839339 7107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:51.839618 7107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:51.839668 7107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:51.844014 7107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:51.844095 7107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:51.844139 7107 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260807 02:21:51.844154 7107 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:21:51.844187 7107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:21:51.850215 7107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.221.126:46169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:51.851333 7107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:51.852331 7107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:51.862212 7113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:51.862257 7112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:51.862251 7115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:51.862571 7107 server_base.cc:1034] running on GCE node
I20260807 02:21:51.863049 7107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:51.863715 7107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:51.864913 7107 hybrid_clock.cc:648] HybridClock initialized: now 1786069311864882 us; error 45 us; skew 500 ppm
Aug 07 02:21:51 dist-test-slave-69rg krb5kdc[6701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069311, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:51.867841 7107 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:21:51.868709 7107 webserver.cc:460] Webserver started at http://127.4.221.67:45103/ using document root <none> and password file <none>
I20260807 02:21:51.869302 7107 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:51.869362 7107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:51.869527 7107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:51.871547 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "012bac788de848698d856e91b895e20a"
format_stamp: "Formatted at 2026-08-07 02:21:51 on dist-test-slave-69rg"
server_key: "475a75fc2d407b35b5abd440dd642391"
server_key_iv: "0d01473677be074d927badbbe3a7fc17"
server_key_version: "encryptionkey@0"
I20260807 02:21:51.872040 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "012bac788de848698d856e91b895e20a"
format_stamp: "Formatted at 2026-08-07 02:21:51 on dist-test-slave-69rg"
server_key: "475a75fc2d407b35b5abd440dd642391"
server_key_iv: "0d01473677be074d927badbbe3a7fc17"
server_key_version: "encryptionkey@0"
I20260807 02:21:51.875995 7107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:21:51.878463 7122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:51.879792 7107 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:21:51.879901 7107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "012bac788de848698d856e91b895e20a"
format_stamp: "Formatted at 2026-08-07 02:21:51 on dist-test-slave-69rg"
server_key: "475a75fc2d407b35b5abd440dd642391"
server_key_iv: "0d01473677be074d927badbbe3a7fc17"
server_key_version: "encryptionkey@0"
I20260807 02:21:51.879993 7107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:51.898416 7107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:51.902272 7107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:51.902448 7107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:51.903951 7107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:21:51.904009 7107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:51.904052 7107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:21:51.904119 7107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:21:51 dist-test-slave-69rg krb5kdc[6701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069311, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:51.915685 7107 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:46219
I20260807 02:21:51.915720 7240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:46219 every 8 connection(s)
I20260807 02:21:51.916671 7107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:21:51.918859 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.904940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:51147 (local address 127.4.221.126:46169)
0807 02:21:51.905086 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:51.905090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:51.906440 (+ 1350us) server_negotiation.cc:407] Connection header received
0807 02:21:51.907670 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:51.907673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:51.907740 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:51.907830 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:51.909361 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.910256 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.911688 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.911932 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.912053 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.914943 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:51.914946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:51.914948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:51.914976 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:51.916613 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.917215 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.917220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.917222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.917337 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.917773 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.917777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.917778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.918051 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:51.918170 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:51.918621 (+ 451us) server_negotiation.cc:299] Negotiation successful
0807 02:21:51.918736 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 02:21:51.919718 7131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.905596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46169 (local address 127.4.221.67:51147)
0807 02:21:51.906230 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:51.906323 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:51.907430 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:51.908030 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:51.908037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:51.908407 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:51.909155 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.909178 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.910379 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.910385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.911551 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.911562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.912123 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.912127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.912349 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.913031 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:51.913048 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:51.914736 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:51.916745 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:51.916751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:51.916765 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:51.917116 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:51.917439 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:51.917442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:51.917444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:51.917637 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:51.918165 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:51.918170 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:51.918410 (+ 240us) client_negotiation.cc:769] Sending connection context
0807 02:21:51.918568 (+ 158us) client_negotiation.cc:240] Negotiation successful
0807 02:21:51.918836 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":189,"threads_started":1}
I20260807 02:21:51.920157 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 7107
I20260807 02:21:51.920289 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:21:51.920679 4981 external_mini_cluster.cc:1428] Setting key 6d705fd6076a511f9f81fe6af74e09bb
I20260807 02:21:51.921134 7241 heartbeater.cc:347] Connected to a master server at 127.4.221.126:46169
I20260807 02:21:51.921294 7241 heartbeater.cc:464] Registering TS with master...
I20260807 02:21:51.921877 7241 heartbeater.cc:511] Master 127.4.221.126:46169 requested a full tablet report, sending...
I20260807 02:21:51.922986 6746 ts_manager.cc:194] Registered new tserver with Master: 012bac788de848698d856e91b895e20a (127.4.221.67:46219)
I20260807 02:21:51.923695 6746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:51147
I20260807 02:21:51.930823 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.923562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:41315 (local address 127.4.221.126:46169)
0807 02:21:51.923699 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:51.923706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:51.923948 (+ 242us) server_negotiation.cc:407] Connection header received
0807 02:21:51.924083 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:51.924086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:51.924137 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:51.924225 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:51.924691 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.925209 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.926110 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.926342 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.926446 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.927603 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:51.927607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:51.927609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:51.927635 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:51.929144 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.929583 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.929587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.929589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.929671 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.929972 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.929976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.929978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.930159 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:51.930270 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:51.930537 (+ 267us) server_negotiation.cc:299] Negotiation successful
0807 02:21:51.930656 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 02:21:51.930845 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.923491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46169 (local address 127.4.221.67:41315)
0807 02:21:51.923852 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:51.923867 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:51.923991 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:51.924243 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:51.924245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:51.924475 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:51.924574 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.924580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.925341 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.925345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.925986 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.925995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.926460 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.926464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.926553 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.927115 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:51.927133 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:51.927460 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:51.929261 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:51.929266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:51.929268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:51.929476 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:51.929763 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:51.929767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:51.929769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:51.929892 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:51.930258 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:51.930261 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:51.930367 (+ 106us) client_negotiation.cc:769] Sending connection context
0807 02:21:51.930478 (+ 111us) client_negotiation.cc:240] Negotiation successful
0807 02:21:51.930640 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":90,"threads_started":1}
I20260807 02:21:51.934963 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 02:21:51 dist-test-slave-69rg krb5kdc[6701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 02:21:51 dist-test-slave-69rg krb5kdc[6701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:51.964110 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.955304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33908 (local address 127.4.221.126:46169)
0807 02:21:51.955477 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:51.955481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:51.955496 (+ 15us) server_negotiation.cc:407] Connection header received
0807 02:21:51.955547 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:51.955549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:51.955591 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:51.955668 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:51.956421 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.957021 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.958058 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.958347 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.958431 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.961025 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:51.961028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:51.961031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:51.961058 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:51.962521 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.962936 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.962938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.962940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.963013 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:51.963310 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:51.963312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:51.963313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:51.963493 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:51.963591 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:51.963803 (+ 212us) server_negotiation.cc:299] Negotiation successful
0807 02:21:51.963905 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 02:21:51.967361 6746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:21:51.970542 6746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:21:51.987527 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.980179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46620 (local address 127.4.221.67:46219)
0807 02:21:51.980596 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:51.980609 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:51.981144 (+ 535us) server_negotiation.cc:407] Connection header received
0807 02:21:51.981491 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:51.981498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:51.981737 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:51.981934 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:51.982592 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.983631 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.985694 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.986415 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.986791 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.986972 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:51.987200 (+ 228us) server_negotiation.cc:299] Negotiation successful
0807 02:21:51.987337 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":156,"threads_started":1}
I20260807 02:21:51.987702 7259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.980858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:39693 (local address 127.0.0.1:55504)
0807 02:21:51.981733 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:51.981746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:51.981877 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:51.982371 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:51.982374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:51.982386 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:51.982600 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.982607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.984099 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.984103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.985551 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.985563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.986992 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.986996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.987234 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.987296 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 02:21:51.987366 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 02:21:51.987547 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":297,"thread_start_us":100,"threads_started":1}
I20260807 02:21:51.988180 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.981321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.4.221.66:39693)
0807 02:21:51.981771 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:51.981804 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:51.981881 (+ 77us) server_negotiation.cc:407] Connection header received
0807 02:21:51.982018 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:51.982023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:51.982202 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:51.982348 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:51.982740 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.983925 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.985715 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.986625 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.986744 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.986933 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:51.987907 (+ 974us) server_negotiation.cc:299] Negotiation successful
0807 02:21:51.988021 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":132,"threads_started":1}
I20260807 02:21:51.988204 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.979997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:46219 (local address 127.0.0.1:46620)
0807 02:21:51.980970 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:51.981035 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:51.981270 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:51.982188 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:51.982191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:51.982233 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:51.982469 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.982476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.983791 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.983794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.985551 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.985563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.986791 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.986794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.986926 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.986976 (+ 50us) client_negotiation.cc:769] Sending connection context
0807 02:21:51.987955 (+ 979us) client_negotiation.cc:240] Negotiation successful
0807 02:21:51.988042 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":188,"spinlock_wait_cycles":3072,"thread_start_us":114,"threads_started":1}
I20260807 02:21:51.989114 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.980171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:35129 (local address 127.0.0.1:52218)
0807 02:21:51.981171 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:51.981186 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:51.981278 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:51.982184 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:51.982190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:51.982229 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:51.982447 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.982457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.984518 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.984522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.985598 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:51.985611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.986992 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.986996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:51.987231 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.987291 (+ 60us) client_negotiation.cc:769] Sending connection context
0807 02:21:51.987366 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 02:21:51.988169 (+ 803us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":770,"spinlock_wait_cycles":6656,"thread_start_us":112,"threads_started":1}
I20260807 02:21:51.989503 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:51.980801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.4.221.65:35129)
0807 02:21:51.981603 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:51.981612 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:51.981639 (+ 27us) server_negotiation.cc:407] Connection header received
0807 02:21:51.981721 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:51.981726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:51.981989 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:51.982155 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:51.982619 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.984278 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.985772 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:51.986625 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:51.986743 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:51.986918 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:51.987529 (+ 611us) server_negotiation.cc:299] Negotiation successful
0807 02:21:51.987640 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":74,"threads_started":1}
I20260807 02:21:51.991314 7174 tablet_service.cc:1467] Processing CreateTablet for tablet acdd2481ce304110b3ba83964d24872b (DEFAULT_TABLE table=test-table [id=44cceddc2e974f7fa882159a0130f135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:51.991382 7030 tablet_service.cc:1467] Processing CreateTablet for tablet acdd2481ce304110b3ba83964d24872b (DEFAULT_TABLE table=test-table [id=44cceddc2e974f7fa882159a0130f135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:51.992455 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdd2481ce304110b3ba83964d24872b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:51.992461 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdd2481ce304110b3ba83964d24872b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:51.992415 6885 tablet_service.cc:1467] Processing CreateTablet for tablet acdd2481ce304110b3ba83964d24872b (DEFAULT_TABLE table=test-table [id=44cceddc2e974f7fa882159a0130f135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:51.993654 6885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acdd2481ce304110b3ba83964d24872b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:52.000245 7264 tablet_bootstrap.cc:492] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a: Bootstrap starting.
I20260807 02:21:52.001315 7265 tablet_bootstrap.cc:492] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375: Bootstrap starting.
I20260807 02:21:52.002370 7266 tablet_bootstrap.cc:492] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c: Bootstrap starting.
I20260807 02:21:52.003590 7265 tablet_bootstrap.cc:654] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:52.004212 7264 tablet_bootstrap.cc:654] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:52.004482 7266 tablet_bootstrap.cc:654] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:52.004582 7265 log.cc:824] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375: Log is configured to *not* fsync() on all Append() calls
I20260807 02:21:52.005188 7264 log.cc:824] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a: Log is configured to *not* fsync() on all Append() calls
I20260807 02:21:52.005285 7266 log.cc:824] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c: Log is configured to *not* fsync() on all Append() calls
I20260807 02:21:52.006798 7265 tablet_bootstrap.cc:492] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375: No bootstrap required, opened a new log
I20260807 02:21:52.007208 7265 ts_tablet_manager.cc:1397] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 02:21:52.007252 7264 tablet_bootstrap.cc:492] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a: No bootstrap required, opened a new log
I20260807 02:21:52.007439 7264 ts_tablet_manager.cc:1397] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260807 02:21:52.008736 7266 tablet_bootstrap.cc:492] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c: No bootstrap required, opened a new log
I20260807 02:21:52.008984 7266 ts_tablet_manager.cc:1397] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260807 02:21:52.010974 7264 raft_consensus.cc:348] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } }
I20260807 02:21:52.011240 7264 raft_consensus.cc:374] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:52.011296 7264 raft_consensus.cc:729] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012bac788de848698d856e91b895e20a, State: Initialized, Role: FOLLOWER
I20260807 02:21:52.011758 7264 consensus_queue.cc:260] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } }
I20260807 02:21:52.011974 7265 raft_consensus.cc:348] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } }
I20260807 02:21:52.012252 7265 raft_consensus.cc:374] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:52.012332 7265 raft_consensus.cc:729] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a98433a0397243a8885c782f775af375, State: Initialized, Role: FOLLOWER
I20260807 02:21:52.012745 7241 heartbeater.cc:503] Master 127.4.221.126:46169 was elected leader, sending a full tablet report...
I20260807 02:21:52.012784 7265 consensus_queue.cc:260] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } }
I20260807 02:21:52.012943 7264 ts_tablet_manager.cc:1428] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 02:21:52.013446 6954 heartbeater.cc:503] Master 127.4.221.126:46169 was elected leader, sending a full tablet report...
I20260807 02:21:52.013753 7265 ts_tablet_manager.cc:1428] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 02:21:52.014716 7266 raft_consensus.cc:348] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } }
I20260807 02:21:52.015050 7266 raft_consensus.cc:374] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:52.015102 7266 raft_consensus.cc:729] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dcfdf2cb750443584931b0a7ab8ae4c, State: Initialized, Role: FOLLOWER
I20260807 02:21:52.015538 7266 consensus_queue.cc:260] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } }
I20260807 02:21:52.016222 7098 heartbeater.cc:503] Master 127.4.221.126:46169 was elected leader, sending a full tablet report...
I20260807 02:21:52.016569 7266 ts_tablet_manager.cc:1428] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260807 02:21:52.052856 7271 raft_consensus.cc:482] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:21:52.053108 7271 raft_consensus.cc:504] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } }
I20260807 02:21:52.054396 7271 leader_election.cc:288] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a98433a0397243a8885c782f775af375 (127.4.221.65:35129), 1dcfdf2cb750443584931b0a7ab8ae4c (127.4.221.66:39693)
I20260807 02:21:52.059131 7131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.054792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:35129 (local address 127.4.221.67:46061)
0807 02:21:52.054950 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:52.054968 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:52.055080 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:52.055469 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:52.055476 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:52.055502 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:52.055642 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.055659 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.056732 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.056735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.057990 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.057999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.058693 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.058697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.058803 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.058832 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 02:21:52.058913 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 02:21:52.058993 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 02:21:52.059237 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.054881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:46061 (local address 127.4.221.65:35129)
0807 02:21:52.055040 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:52.055046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:52.055061 (+ 15us) server_negotiation.cc:407] Connection header received
0807 02:21:52.055211 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:52.055217 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:52.055271 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:52.055375 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:52.055786 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.056631 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.058108 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.058566 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.058704 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.058781 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:52.058949 (+ 168us) server_negotiation.cc:299] Negotiation successful
0807 02:21:52.059089 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 02:21:52.059940 6906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdd2481ce304110b3ba83964d24872b" candidate_uuid: "012bac788de848698d856e91b895e20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a98433a0397243a8885c782f775af375" is_pre_election: true
I20260807 02:21:52.060227 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.055125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:39693 (local address 127.4.221.67:52827)
0807 02:21:52.055530 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:52.055543 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:52.055626 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:52.055869 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:52.055871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:52.055888 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:52.056437 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.056446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.057551 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.057554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.058780 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.058800 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.059735 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.059738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.059843 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.059885 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 02:21:52.059988 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 02:21:52.060062 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":49,"thread_start_us":161,"threads_started":1}
I20260807 02:21:52.060344 6906 raft_consensus.cc:2454] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012bac788de848698d856e91b895e20a in term 0.
I20260807 02:21:52.060807 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.055314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:52827 (local address 127.4.221.66:39693)
0807 02:21:52.055454 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:52.055459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:52.055627 (+ 168us) server_negotiation.cc:407] Connection header received
0807 02:21:52.055704 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:52.055708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:52.055767 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:52.055835 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:52.056619 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.057448 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.058891 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.059516 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.059619 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.059663 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:52.060042 (+ 379us) server_negotiation.cc:299] Negotiation successful
0807 02:21:52.060643 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:21:52.060925 7123 leader_election.cc:302] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012bac788de848698d856e91b895e20a, a98433a0397243a8885c782f775af375; no voters:
I20260807 02:21:52.061231 7271 raft_consensus.cc:2781] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:21:52.061323 7271 raft_consensus.cc:482] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:21:52.061379 7271 raft_consensus.cc:3037] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:52.061517 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdd2481ce304110b3ba83964d24872b" candidate_uuid: "012bac788de848698d856e91b895e20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" is_pre_election: true
I20260807 02:21:52.061853 7052 raft_consensus.cc:2454] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012bac788de848698d856e91b895e20a in term 0.
I20260807 02:21:52.063014 7271 raft_consensus.cc:504] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } }
I20260807 02:21:52.063406 7271 leader_election.cc:288] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [CANDIDATE]: Term 1 election: Requested vote from peers a98433a0397243a8885c782f775af375 (127.4.221.65:35129), 1dcfdf2cb750443584931b0a7ab8ae4c (127.4.221.66:39693)
I20260807 02:21:52.063910 6906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdd2481ce304110b3ba83964d24872b" candidate_uuid: "012bac788de848698d856e91b895e20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a98433a0397243a8885c782f775af375"
I20260807 02:21:52.063963 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acdd2481ce304110b3ba83964d24872b" candidate_uuid: "012bac788de848698d856e91b895e20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c"
I20260807 02:21:52.064085 6906 raft_consensus.cc:3037] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:52.064110 7052 raft_consensus.cc:3037] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:52.065500 7052 raft_consensus.cc:2454] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012bac788de848698d856e91b895e20a in term 1.
I20260807 02:21:52.065500 6906 raft_consensus.cc:2454] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012bac788de848698d856e91b895e20a in term 1.
I20260807 02:21:52.065879 7123 leader_election.cc:302] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012bac788de848698d856e91b895e20a, 1dcfdf2cb750443584931b0a7ab8ae4c; no voters:
I20260807 02:21:52.066218 7271 raft_consensus.cc:2781] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:21:52.066468 7271 raft_consensus.cc:686] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [term 1 LEADER]: Becoming Leader. State: Replica: 012bac788de848698d856e91b895e20a, State: Running, Role: LEADER
I20260807 02:21:52.066843 7271 consensus_queue.cc:237] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } }
I20260807 02:21:52.070271 6744 catalog_manager.cc:5495] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a reported cstate change: term changed from 0 to 1, leader changed from <none> to 012bac788de848698d856e91b895e20a (127.4.221.67). New cstate: current_term: 1 leader_uuid: "012bac788de848698d856e91b895e20a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } health_report { overall_health: UNKNOWN } } }
Aug 07 02:21:52 dist-test-slave-69rg krb5kdc[6701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069311, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:52.134352 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.123918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46169 (local address 127.0.0.1:33918)
0807 02:21:52.124327 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:52.124343 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:52.124464 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:52.124836 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:52.124838 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:52.125273 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:52.125391 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.125398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.126431 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.126434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.127009 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.127017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.127614 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.127617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.127698 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.128373 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:52.128386 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:52.130286 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:52.132400 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:52.132411 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:52.132417 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:52.132748 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:52.133146 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:52.133149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:52.133151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:52.133248 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:52.133647 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:52.133653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:52.133804 (+ 151us) client_negotiation.cc:769] Sending connection context
0807 02:21:52.133982 (+ 178us) client_negotiation.cc:240] Negotiation successful
0807 02:21:52.134166 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":174,"threads_started":1}
I20260807 02:21:52.134400 6796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.124006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.4.221.126:46169)
0807 02:21:52.124223 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:52.124229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:52.124435 (+ 206us) server_negotiation.cc:407] Connection header received
0807 02:21:52.124603 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:52.124606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:52.124662 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:52.124764 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:52.125576 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.126197 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.127162 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.127424 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.127524 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.130451 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:52.130455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:52.130456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:52.130481 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:52.132287 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:52.132849 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:52.132852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:52.132853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:52.132929 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:52.133341 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:52.133343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:52.133344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:52.133543 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:52.133622 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:52.134067 (+ 445us) server_negotiation.cc:299] Negotiation successful
0807 02:21:52.134285 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":83}
I20260807 02:21:52.138943 6744 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:33918:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 02:21:52.139735 6744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:21:52.145011 7030 tablet_service.cc:1467] Processing CreateTablet for tablet 74a5dad47e88416f83e431e4e2833ca4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f95807af50914bd289ec3b4fa08c27e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:21:52.145112 7174 tablet_service.cc:1467] Processing CreateTablet for tablet 74a5dad47e88416f83e431e4e2833ca4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f95807af50914bd289ec3b4fa08c27e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:21:52.145030 6885 tablet_service.cc:1467] Processing CreateTablet for tablet 74a5dad47e88416f83e431e4e2833ca4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f95807af50914bd289ec3b4fa08c27e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:21:52.145714 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a5dad47e88416f83e431e4e2833ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:52.145733 6885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a5dad47e88416f83e431e4e2833ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:52.145733 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a5dad47e88416f83e431e4e2833ca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:52.148772 7264 tablet_bootstrap.cc:492] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a: Bootstrap starting.
I20260807 02:21:52.149121 7265 tablet_bootstrap.cc:492] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375: Bootstrap starting.
I20260807 02:21:52.149806 7264 tablet_bootstrap.cc:654] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:52.150005 7266 tablet_bootstrap.cc:492] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c: Bootstrap starting.
I20260807 02:21:52.150127 7265 tablet_bootstrap.cc:654] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:52.151129 7266 tablet_bootstrap.cc:654] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:52.151588 7265 tablet_bootstrap.cc:492] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375: No bootstrap required, opened a new log
I20260807 02:21:52.151674 7265 ts_tablet_manager.cc:1397] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 02:21:52.152169 7265 raft_consensus.cc:348] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.152351 7264 tablet_bootstrap.cc:492] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a: No bootstrap required, opened a new log
I20260807 02:21:52.152382 7265 raft_consensus.cc:374] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:52.152434 7265 raft_consensus.cc:729] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a98433a0397243a8885c782f775af375, State: Initialized, Role: FOLLOWER
I20260807 02:21:52.152452 7264 ts_tablet_manager.cc:1397] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 02:21:52.152572 7265 consensus_queue.cc:260] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.152839 7265 ts_tablet_manager.cc:1428] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 02:21:52.152884 7272 tablet_replica.cc:439] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } }
I20260807 02:21:52.152956 7264 raft_consensus.cc:348] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.153059 7272 tablet_replica.cc:442] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:21:52.153057 7264 raft_consensus.cc:374] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:52.153100 7264 raft_consensus.cc:729] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012bac788de848698d856e91b895e20a, State: Initialized, Role: FOLLOWER
I20260807 02:21:52.153247 7264 consensus_queue.cc:260] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.153512 7264 ts_tablet_manager.cc:1428] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260807 02:21:52.153570 7271 tablet_replica.cc:439] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } }
I20260807 02:21:52.153680 7266 tablet_bootstrap.cc:492] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c: No bootstrap required, opened a new log
I20260807 02:21:52.153704 7271 tablet_replica.cc:442] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:21:52.153779 7266 ts_tablet_manager.cc:1397] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 02:21:52.154335 7266 raft_consensus.cc:348] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.154452 7266 raft_consensus.cc:374] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:52.154486 7266 raft_consensus.cc:729] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dcfdf2cb750443584931b0a7ab8ae4c, State: Initialized, Role: FOLLOWER
I20260807 02:21:52.154649 7266 consensus_queue.cc:260] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.154953 7266 ts_tablet_manager.cc:1428] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 02:21:52.154937 7273 tablet_replica.cc:439] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } }
I20260807 02:21:52.155092 7273 tablet_replica.cc:442] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c: This TxnStatusTablet replica's current role is: FOLLOWER
W20260807 02:21:52.167745 7242 tablet.cc:2369] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:21:52.178864 7099 tablet.cc:2369] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:21:52.184548 6955 tablet.cc:2369] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:21:52.296568 7273 raft_consensus.cc:482] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:21:52.296805 7273 raft_consensus.cc:504] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.298391 7273 leader_election.cc:288] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a98433a0397243a8885c782f775af375 (127.4.221.65:35129), 012bac788de848698d856e91b895e20a (127.4.221.67:46219)
I20260807 02:21:52.303408 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.298970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:50923 (local address 127.4.221.65:35129)
0807 02:21:52.299147 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:52.299151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:52.299210 (+ 59us) server_negotiation.cc:407] Connection header received
0807 02:21:52.299433 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:52.299437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:52.299556 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:52.299648 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:52.300025 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.300672 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.302021 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.302723 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.302864 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.302926 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:52.303172 (+ 246us) server_negotiation.cc:299] Negotiation successful
0807 02:21:52.303274 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 02:21:52.303447 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.298890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:45285 (local address 127.4.221.67:46219)
0807 02:21:52.299093 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:52.299100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:52.299634 (+ 534us) server_negotiation.cc:407] Connection header received
0807 02:21:52.299725 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:52.299728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:52.299796 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:52.299882 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:52.300191 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.300697 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.302114 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.302746 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.302864 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.302926 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:52.303234 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 02:21:52.303330 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260807 02:21:52.303469 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.298731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:46219 (local address 127.4.221.66:45285)
0807 02:21:52.299497 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:52.299512 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:52.299631 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:52.299902 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:52.299905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:52.299920 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:52.300081 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.300088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.300866 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.300869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.301921 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.301934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.302996 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.302999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.303094 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.303127 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 02:21:52.303205 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 02:21:52.303290 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":391,"thread_start_us":113,"threads_started":1}
I20260807 02:21:52.303858 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.298731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:35129 (local address 127.4.221.66:50923)
0807 02:21:52.299092 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:52.299111 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:52.299278 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:52.299687 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:52.299690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:52.299714 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:52.299884 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.299892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.300798 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.300802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.301890 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.301906 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.302864 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.302868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.302994 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.303023 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 02:21:52.303607 (+ 584us) client_negotiation.cc:240] Negotiation successful
0807 02:21:52.303719 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":51,"thread_start_us":142,"threads_started":1}
I20260807 02:21:52.304250 7194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a5dad47e88416f83e431e4e2833ca4" candidate_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012bac788de848698d856e91b895e20a" is_pre_election: true
I20260807 02:21:52.304378 6906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a5dad47e88416f83e431e4e2833ca4" candidate_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a98433a0397243a8885c782f775af375" is_pre_election: true
I20260807 02:21:52.304571 6906 raft_consensus.cc:2454] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dcfdf2cb750443584931b0a7ab8ae4c in term 0.
I20260807 02:21:52.304584 7194 raft_consensus.cc:2454] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dcfdf2cb750443584931b0a7ab8ae4c in term 0.
I20260807 02:21:52.305011 6980 leader_election.cc:302] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dcfdf2cb750443584931b0a7ab8ae4c, a98433a0397243a8885c782f775af375; no voters:
I20260807 02:21:52.305357 7273 raft_consensus.cc:2781] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:21:52.305455 7273 raft_consensus.cc:482] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:21:52.305507 7273 raft_consensus.cc:3037] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:52.306830 7273 raft_consensus.cc:504] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.307272 7273 leader_election.cc:288] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [CANDIDATE]: Term 1 election: Requested vote from peers a98433a0397243a8885c782f775af375 (127.4.221.65:35129), 012bac788de848698d856e91b895e20a (127.4.221.67:46219)
I20260807 02:21:52.307650 6906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a5dad47e88416f83e431e4e2833ca4" candidate_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a98433a0397243a8885c782f775af375"
I20260807 02:21:52.307695 7194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a5dad47e88416f83e431e4e2833ca4" candidate_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012bac788de848698d856e91b895e20a"
I20260807 02:21:52.307835 6906 raft_consensus.cc:3037] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:52.307839 7194 raft_consensus.cc:3037] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:52.308977 6906 raft_consensus.cc:2454] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dcfdf2cb750443584931b0a7ab8ae4c in term 1.
I20260807 02:21:52.309062 7194 raft_consensus.cc:2454] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dcfdf2cb750443584931b0a7ab8ae4c in term 1.
I20260807 02:21:52.309291 6980 leader_election.cc:302] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dcfdf2cb750443584931b0a7ab8ae4c, a98433a0397243a8885c782f775af375; no voters:
I20260807 02:21:52.309528 7273 raft_consensus.cc:2781] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:21:52.309753 7273 raft_consensus.cc:686] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 LEADER]: Becoming Leader. State: Replica: 1dcfdf2cb750443584931b0a7ab8ae4c, State: Running, Role: LEADER
I20260807 02:21:52.310195 7273 consensus_queue.cc:237] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.313190 7288 tablet_replica.cc:439] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c: TxnStatusTablet state changed. Reason: New leader 1dcfdf2cb750443584931b0a7ab8ae4c. Latest consensus state: current_term: 1 leader_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } }
I20260807 02:21:52.313328 7288 tablet_replica.cc:442] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:21:52.313654 7290 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 02:21:52.313896 7290 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 02:21:52.314926 6746 catalog_manager.cc:5495] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dcfdf2cb750443584931b0a7ab8ae4c (127.4.221.66). New cstate: current_term: 1 leader_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } health_report { overall_health: UNKNOWN } } }
I20260807 02:21:52.352101 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.347914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:39693 (local address 127.0.0.1:55520)
0807 02:21:52.348089 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:52.348110 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:52.348238 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:52.348638 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:52.348641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:52.348650 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:21:52.348789 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.348796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.349589 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.349594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.350275 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.350283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.350757 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.350761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.350871 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.350920 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:21:52.351754 (+ 834us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:21:52.351772 (+ 18us) client_negotiation.cc:769] Sending connection context
0807 02:21:52.351848 (+ 76us) client_negotiation.cc:240] Negotiation successful
0807 02:21:52.351961 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260807 02:21:52.352104 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.348045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.4.221.66:39693)
0807 02:21:52.348208 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:52.348213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:52.348231 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:21:52.348439 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:52.348442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:52.348516 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:52.348624 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:21:52.348909 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.349466 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.350395 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.350635 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.350748 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.351108 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:21:52.351651 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:21:52.351793 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:52.351859 (+ 66us) server_negotiation.cc:299] Negotiation successful
0807 02:21:52.351923 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 02:21:52.358946 6906 raft_consensus.cc:1261] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Refusing update from remote peer 1dcfdf2cb750443584931b0a7ab8ae4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:52.358942 7194 raft_consensus.cc:1261] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a [term 1 FOLLOWER]: Refusing update from remote peer 1dcfdf2cb750443584931b0a7ab8ae4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:52.359824 7273 consensus_queue.cc:1035] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:21:52.360046 7288 consensus_queue.cc:1035] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:21:52.364187 7271 tablet_replica.cc:439] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a: TxnStatusTablet state changed. Reason: New leader 1dcfdf2cb750443584931b0a7ab8ae4c. Latest consensus state: current_term: 1 leader_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } }
I20260807 02:21:52.364322 7271 tablet_replica.cc:442] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:21:52.365863 7272 tablet_replica.cc:439] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375: TxnStatusTablet state changed. Reason: New leader 1dcfdf2cb750443584931b0a7ab8ae4c. Latest consensus state: current_term: 1 leader_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } }
I20260807 02:21:52.365994 7272 tablet_replica.cc:442] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:21:52.369102 7288 tablet_replica.cc:439] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } }
I20260807 02:21:52.369252 7288 tablet_replica.cc:442] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:21:52.369388 7288 tablet_replica.cc:439] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } }
I20260807 02:21:52.369486 7288 tablet_replica.cc:442] T 74a5dad47e88416f83e431e4e2833ca4 P 1dcfdf2cb750443584931b0a7ab8ae4c: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:21:52.369863 7293 mvcc.cc:205] Tried to move back new op lower bound from 7315739903412834304 to 7315739903226056704. Current Snapshot: MvccSnapshot[applied={T|T < 7315739903412834304}]
I20260807 02:21:52.371814 7294 mvcc.cc:205] Tried to move back new op lower bound from 7315739903412834304 to 7315739903226056704. Current Snapshot: MvccSnapshot[applied={T|T < 7315739903412834304}]
I20260807 02:21:52.372014 7272 tablet_replica.cc:439] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } }
I20260807 02:21:52.372123 7272 tablet_replica.cc:442] T 74a5dad47e88416f83e431e4e2833ca4 P a98433a0397243a8885c782f775af375: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:21:52.372429 7271 tablet_replica.cc:439] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } } }
I20260807 02:21:52.372570 7271 tablet_replica.cc:442] T 74a5dad47e88416f83e431e4e2833ca4 P 012bac788de848698d856e91b895e20a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:21:52.384691 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.379805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46628 (local address 127.4.221.67:46219)
0807 02:21:52.379940 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:52.379945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:52.379961 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:21:52.380164 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:52.380168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:52.380243 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:52.380362 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:21:52.380659 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.381478 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.382685 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.383042 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.383150 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.383593 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:21:52.384207 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:21:52.384375 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:52.384424 (+ 49us) server_negotiation.cc:299] Negotiation successful
0807 02:21:52.384491 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:21:52.398988 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.394942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:59279 (local address 127.4.221.66:39693)
0807 02:21:52.395119 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:52.395125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:52.395144 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:21:52.395280 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:52.395284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:52.395351 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:52.395435 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:21:52.395703 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.396580 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.397516 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.397908 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.398017 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.398388 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:21:52.398495 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:21:52.398579 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:52.398745 (+ 166us) server_negotiation.cc:299] Negotiation successful
0807 02:21:52.398814 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 02:21:52.398994 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.394786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:39693 (local address 127.4.221.67:59279)
0807 02:21:52.395050 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:52.395071 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:52.395164 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:52.395453 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:52.395455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:52.395462 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:21:52.395592 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.395596 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.396716 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.396719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.397385 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.397394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.398039 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.398042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.398140 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.398197 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:21:52.398624 (+ 427us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:21:52.398632 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 02:21:52.398715 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 02:21:52.398813 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260807 02:21:52.409164 7052 raft_consensus.cc:1261] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Refusing update from remote peer 012bac788de848698d856e91b895e20a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:52.409926 6905 raft_consensus.cc:1261] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Refusing update from remote peer 012bac788de848698d856e91b895e20a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:52.410034 7275 consensus_queue.cc:1035] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:21:52.410372 7271 consensus_queue.cc:1035] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:21:52.418362 7295 mvcc.cc:205] Tried to move back new op lower bound from 7315739903620235264 to 7315739902229127168. Current Snapshot: MvccSnapshot[applied={T|T < 7315739903620235264}]
I20260807 02:21:52.444056 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.438290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:60989 (local address 127.4.221.67:46219)
0807 02:21:52.438442 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:52.438449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:52.438847 (+ 398us) server_negotiation.cc:407] Connection header received
0807 02:21:52.439037 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:52.439040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:52.439109 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:52.439202 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:21:52.440273 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.441096 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.442517 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.442835 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.442943 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.443373 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:21:52.443483 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:21:52.443554 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:52.443793 (+ 239us) server_negotiation.cc:299] Negotiation successful
0807 02:21:52.443869 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 02:21:52.444952 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:52.438234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:46219 (local address 127.4.221.66:60989)
0807 02:21:52.438744 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:52.438759 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:52.438912 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:52.439216 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:52.439452 (+ 236us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:52.439472 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:21:52.439720 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.439731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.441400 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.441404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.442323 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:52.442334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:52.442975 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:52.442979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:52.443094 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:52.443173 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:21:52.443678 (+ 505us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:21:52.443687 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 02:21:52.444613 (+ 926us) client_negotiation.cc:240] Negotiation successful
0807 02:21:52.444730 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":183,"threads_started":1}
I20260807 02:21:52.483712 6744 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:21:52.483997 6744 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:21:52.487528 6744 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5: Sending DeleteTablet for 3 replicas of tablet acdd2481ce304110b3ba83964d24872b
I20260807 02:21:52.488479 7174 tablet_service.cc:1514] Processing DeleteTablet for tablet acdd2481ce304110b3ba83964d24872b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:52 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:46620
I20260807 02:21:52.488998 6885 tablet_service.cc:1514] Processing DeleteTablet for tablet acdd2481ce304110b3ba83964d24872b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:52 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:52218
I20260807 02:21:52.489487 7313 tablet_replica.cc:330] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a: stopping tablet replica
I20260807 02:21:52.489619 7030 tablet_service.cc:1514] Processing DeleteTablet for tablet acdd2481ce304110b3ba83964d24872b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:52 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:55504
I20260807 02:21:52.490074 7313 raft_consensus.cc:2229] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:21:52.490607 7313 raft_consensus.cc:2258] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:52.491430 6744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:21:52.492017 6744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:21:52.492893 7313 ts_tablet_manager.cc:1905] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:21:52.494035 7314 tablet_replica.cc:330] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375: stopping tablet replica
I20260807 02:21:52.494472 7314 raft_consensus.cc:2229] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:21:52.494735 7314 raft_consensus.cc:2258] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:52.498247 7174 tablet_service.cc:1467] Processing CreateTablet for tablet 57a084990a1b49efad6ba87a5f10dcdb (DEFAULT_TABLE table=test-table [id=286a029f0b824c7cb4b9448ee02c5485]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:52.498369 6885 tablet_service.cc:1467] Processing CreateTablet for tablet 57a084990a1b49efad6ba87a5f10dcdb (DEFAULT_TABLE table=test-table [id=286a029f0b824c7cb4b9448ee02c5485]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:52.498610 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a084990a1b49efad6ba87a5f10dcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:52.498709 6885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a084990a1b49efad6ba87a5f10dcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:52.501417 7264 tablet_bootstrap.cc:492] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a: Bootstrap starting.
I20260807 02:21:52.501785 7030 tablet_service.cc:1467] Processing CreateTablet for tablet 57a084990a1b49efad6ba87a5f10dcdb (DEFAULT_TABLE table=test-table [id=286a029f0b824c7cb4b9448ee02c5485]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:52.502154 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57a084990a1b49efad6ba87a5f10dcdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:52.502620 7264 tablet_bootstrap.cc:654] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:52.505187 7313 ts_tablet_manager.cc:1918] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:21:52.505358 7265 tablet_bootstrap.cc:492] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375: Bootstrap starting.
I20260807 02:21:52.505519 7266 tablet_bootstrap.cc:492] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c: Bootstrap starting.
I20260807 02:21:52.505775 7315 tablet_replica.cc:330] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c: stopping tablet replica
I20260807 02:21:52.506394 7315 raft_consensus.cc:2229] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:21:52.506606 7265 tablet_bootstrap.cc:654] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:52.506606 7266 tablet_bootstrap.cc:654] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:52.507344 7315 raft_consensus.cc:2258] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:52.507939 7314 ts_tablet_manager.cc:1905] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:21:52.509945 7313 log.cc:1196] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/acdd2481ce304110b3ba83964d24872b
I20260807 02:21:52.510396 7313 ts_tablet_manager.cc:1939] T acdd2481ce304110b3ba83964d24872b P 012bac788de848698d856e91b895e20a: Deleting consensus metadata
I20260807 02:21:52.510437 7315 ts_tablet_manager.cc:1905] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:21:52.512001 6731 catalog_manager.cc:4841] TS 012bac788de848698d856e91b895e20a (127.4.221.67:46219): tablet acdd2481ce304110b3ba83964d24872b (table test-table [id=44cceddc2e974f7fa882159a0130f135]) successfully deleted
I20260807 02:21:52.513371 7264 tablet_bootstrap.cc:492] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a: No bootstrap required, opened a new log
I20260807 02:21:52.513484 7264 ts_tablet_manager.cc:1397] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260807 02:21:52.514211 7314 ts_tablet_manager.cc:1918] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:21:52.514156 7264 raft_consensus.cc:348] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.514302 7264 raft_consensus.cc:374] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:52.514340 7264 raft_consensus.cc:729] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012bac788de848698d856e91b895e20a, State: Initialized, Role: FOLLOWER
I20260807 02:21:52.514480 7264 consensus_queue.cc:260] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.514539 7314 log.cc:1196] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/wal/wals/acdd2481ce304110b3ba83964d24872b
I20260807 02:21:52.514848 7264 ts_tablet_manager.cc:1428] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 02:21:52.515093 7314 ts_tablet_manager.cc:1939] T acdd2481ce304110b3ba83964d24872b P a98433a0397243a8885c782f775af375: Deleting consensus metadata
I20260807 02:21:52.515133 7265 tablet_bootstrap.cc:492] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375: No bootstrap required, opened a new log
I20260807 02:21:52.515264 7265 ts_tablet_manager.cc:1397] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260807 02:21:52.515774 7265 raft_consensus.cc:348] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.515885 7265 raft_consensus.cc:374] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:52.515914 7265 raft_consensus.cc:729] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a98433a0397243a8885c782f775af375, State: Initialized, Role: FOLLOWER
I20260807 02:21:52.516040 7265 consensus_queue.cc:260] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.516391 7265 ts_tablet_manager.cc:1428] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 02:21:52.516607 7315 ts_tablet_manager.cc:1918] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:21:52.516727 7315 log.cc:1196] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/acdd2481ce304110b3ba83964d24872b
I20260807 02:21:52.517076 7315 ts_tablet_manager.cc:1939] T acdd2481ce304110b3ba83964d24872b P 1dcfdf2cb750443584931b0a7ab8ae4c: Deleting consensus metadata
I20260807 02:21:52.518657 6730 catalog_manager.cc:4841] TS 1dcfdf2cb750443584931b0a7ab8ae4c (127.4.221.66:39693): tablet acdd2481ce304110b3ba83964d24872b (table test-table [id=44cceddc2e974f7fa882159a0130f135]) successfully deleted
I20260807 02:21:52.518985 7266 tablet_bootstrap.cc:492] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c: No bootstrap required, opened a new log
I20260807 02:21:52.519040 6730 catalog_manager.cc:4841] TS a98433a0397243a8885c782f775af375 (127.4.221.65:35129): tablet acdd2481ce304110b3ba83964d24872b (table test-table [id=44cceddc2e974f7fa882159a0130f135]) successfully deleted
I20260807 02:21:52.519083 7266 ts_tablet_manager.cc:1397] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260807 02:21:52.519577 7266 raft_consensus.cc:348] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.519750 7266 raft_consensus.cc:374] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:52.519785 7266 raft_consensus.cc:729] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dcfdf2cb750443584931b0a7ab8ae4c, State: Initialized, Role: FOLLOWER
I20260807 02:21:52.519932 7266 consensus_queue.cc:260] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.520237 7266 ts_tablet_manager.cc:1428] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 02:21:52.618141 7275 raft_consensus.cc:482] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:21:52.618379 7275 raft_consensus.cc:504] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.619017 7275 leader_election.cc:288] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a98433a0397243a8885c782f775af375 (127.4.221.65:35129), 1dcfdf2cb750443584931b0a7ab8ae4c (127.4.221.66:39693)
I20260807 02:21:52.619757 6905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a084990a1b49efad6ba87a5f10dcdb" candidate_uuid: "012bac788de848698d856e91b895e20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a98433a0397243a8885c782f775af375" is_pre_election: true
I20260807 02:21:52.619796 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a084990a1b49efad6ba87a5f10dcdb" candidate_uuid: "012bac788de848698d856e91b895e20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" is_pre_election: true
I20260807 02:21:52.619978 6905 raft_consensus.cc:2454] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012bac788de848698d856e91b895e20a in term 0.
I20260807 02:21:52.620044 7052 raft_consensus.cc:2454] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012bac788de848698d856e91b895e20a in term 0.
I20260807 02:21:52.620398 7123 leader_election.cc:302] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012bac788de848698d856e91b895e20a, a98433a0397243a8885c782f775af375; no voters:
I20260807 02:21:52.620710 7275 raft_consensus.cc:2781] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:21:52.620805 7275 raft_consensus.cc:482] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:21:52.620862 7275 raft_consensus.cc:3037] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:52.622378 7275 raft_consensus.cc:504] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.622784 7275 leader_election.cc:288] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [CANDIDATE]: Term 1 election: Requested vote from peers a98433a0397243a8885c782f775af375 (127.4.221.65:35129), 1dcfdf2cb750443584931b0a7ab8ae4c (127.4.221.66:39693)
I20260807 02:21:52.623328 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a084990a1b49efad6ba87a5f10dcdb" candidate_uuid: "012bac788de848698d856e91b895e20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c"
I20260807 02:21:52.623356 6905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57a084990a1b49efad6ba87a5f10dcdb" candidate_uuid: "012bac788de848698d856e91b895e20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a98433a0397243a8885c782f775af375"
I20260807 02:21:52.623520 6905 raft_consensus.cc:3037] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:52.623513 7052 raft_consensus.cc:3037] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:52.624890 7052 raft_consensus.cc:2454] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012bac788de848698d856e91b895e20a in term 1.
I20260807 02:21:52.625226 6905 raft_consensus.cc:2454] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012bac788de848698d856e91b895e20a in term 1.
I20260807 02:21:52.625272 7123 leader_election.cc:302] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012bac788de848698d856e91b895e20a, 1dcfdf2cb750443584931b0a7ab8ae4c; no voters:
I20260807 02:21:52.625500 7275 raft_consensus.cc:2781] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:21:52.625643 7275 raft_consensus.cc:686] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [term 1 LEADER]: Becoming Leader. State: Replica: 012bac788de848698d856e91b895e20a, State: Running, Role: LEADER
I20260807 02:21:52.625871 7275 consensus_queue.cc:237] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:52.628230 6744 catalog_manager.cc:5495] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a reported cstate change: term changed from 0 to 1, leader changed from <none> to 012bac788de848698d856e91b895e20a (127.4.221.67). New cstate: current_term: 1 leader_uuid: "012bac788de848698d856e91b895e20a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } health_report { overall_health: HEALTHY } } }
I20260807 02:21:52.651912 6905 raft_consensus.cc:1261] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Refusing update from remote peer 012bac788de848698d856e91b895e20a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:52.652136 7052 raft_consensus.cc:1261] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Refusing update from remote peer 012bac788de848698d856e91b895e20a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:52.652621 7271 consensus_queue.cc:1035] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:21:52.652839 7275 consensus_queue.cc:1035] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 02:21:52.669729 7295 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 02:21:54.235599 6744 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:21:54.235808 6744 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:21:54.238963 6744 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5: Sending DeleteTablet for 3 replicas of tablet 57a084990a1b49efad6ba87a5f10dcdb
I20260807 02:21:54.239725 7030 tablet_service.cc:1514] Processing DeleteTablet for tablet 57a084990a1b49efad6ba87a5f10dcdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:54 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:55504
I20260807 02:21:54.239753 7174 tablet_service.cc:1514] Processing DeleteTablet for tablet 57a084990a1b49efad6ba87a5f10dcdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:54 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:46620
I20260807 02:21:54.239781 6885 tablet_service.cc:1514] Processing DeleteTablet for tablet 57a084990a1b49efad6ba87a5f10dcdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:54 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:52218
I20260807 02:21:54.240206 7347 tablet_replica.cc:330] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375: stopping tablet replica
I20260807 02:21:54.240381 7347 raft_consensus.cc:2229] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:21:54.240543 7347 raft_consensus.cc:2258] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:54.241241 7347 ts_tablet_manager.cc:1905] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:21:54.244571 7345 tablet_replica.cc:330] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c: stopping tablet replica
I20260807 02:21:54.244763 7345 raft_consensus.cc:2229] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:21:54.244791 7347 ts_tablet_manager.cc:1918] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 02:21:54.244884 7347 log.cc:1196] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/wal/wals/57a084990a1b49efad6ba87a5f10dcdb
I20260807 02:21:54.244942 7345 raft_consensus.cc:2258] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:54.245208 7347 ts_tablet_manager.cc:1939] T 57a084990a1b49efad6ba87a5f10dcdb P a98433a0397243a8885c782f775af375: Deleting consensus metadata
I20260807 02:21:54.245260 7346 tablet_replica.cc:330] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a: stopping tablet replica
I20260807 02:21:54.245438 7346 raft_consensus.cc:2229] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:21:54.245710 7346 raft_consensus.cc:2258] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:54.245801 7345 ts_tablet_manager.cc:1905] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:21:54.246078 6730 catalog_manager.cc:4841] TS a98433a0397243a8885c782f775af375 (127.4.221.65:35129): tablet 57a084990a1b49efad6ba87a5f10dcdb (table test-table [id=286a029f0b824c7cb4b9448ee02c5485]) successfully deleted
I20260807 02:21:54.249709 7345 ts_tablet_manager.cc:1918] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 02:21:54.249847 7345 log.cc:1196] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/57a084990a1b49efad6ba87a5f10dcdb
I20260807 02:21:54.249979 7346 ts_tablet_manager.cc:1905] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:21:54.250177 7345 ts_tablet_manager.cc:1939] T 57a084990a1b49efad6ba87a5f10dcdb P 1dcfdf2cb750443584931b0a7ab8ae4c: Deleting consensus metadata
I20260807 02:21:54.251197 6730 catalog_manager.cc:4841] TS 1dcfdf2cb750443584931b0a7ab8ae4c (127.4.221.66:39693): tablet 57a084990a1b49efad6ba87a5f10dcdb (table test-table [id=286a029f0b824c7cb4b9448ee02c5485]) successfully deleted
I20260807 02:21:54.253541 7346 ts_tablet_manager.cc:1918] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 02:21:54.253645 7346 log.cc:1196] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/57a084990a1b49efad6ba87a5f10dcdb
I20260807 02:21:54.253995 7346 ts_tablet_manager.cc:1939] T 57a084990a1b49efad6ba87a5f10dcdb P 012bac788de848698d856e91b895e20a: Deleting consensus metadata
I20260807 02:21:54.254895 6731 catalog_manager.cc:4841] TS 012bac788de848698d856e91b895e20a (127.4.221.67:46219): tablet 57a084990a1b49efad6ba87a5f10dcdb (table test-table [id=286a029f0b824c7cb4b9448ee02c5485]) successfully deleted
Aug 07 02:21:54 dist-test-slave-69rg krb5kdc[6701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069314, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 02:21:54 dist-test-slave-69rg krb5kdc[6701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069314, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:54.278780 7355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:54.268069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33930 (local address 127.4.221.126:46169)
0807 02:21:54.268542 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:54.268551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:54.268609 (+ 58us) server_negotiation.cc:407] Connection header received
0807 02:21:54.268674 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:54.268677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:54.268739 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:54.268847 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:54.269763 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.270577 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.271630 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.271900 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.271995 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:54.275074 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:54.275077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:54.275080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:54.275111 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:54.276876 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:54.277308 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:54.277310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:54.277312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:54.277432 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:54.277669 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:54.277671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:54.277672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:54.277966 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:54.278084 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:54.278456 (+ 372us) server_negotiation.cc:299] Negotiation successful
0807 02:21:54.278616 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":180,"threads_started":1}
I20260807 02:21:54.282011 6744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:21:54.282609 6744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:21:54.288244 7030 tablet_service.cc:1467] Processing CreateTablet for tablet 0b692610b37d43a4aff572d6156b300b (DEFAULT_TABLE table=test-table [id=66709c1e13a3421c9c145096a115379f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:54.288225 6885 tablet_service.cc:1467] Processing CreateTablet for tablet 0b692610b37d43a4aff572d6156b300b (DEFAULT_TABLE table=test-table [id=66709c1e13a3421c9c145096a115379f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:54.288275 7174 tablet_service.cc:1467] Processing CreateTablet for tablet 0b692610b37d43a4aff572d6156b300b (DEFAULT_TABLE table=test-table [id=66709c1e13a3421c9c145096a115379f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:54.288633 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b692610b37d43a4aff572d6156b300b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:54.288636 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b692610b37d43a4aff572d6156b300b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:54.288636 6885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b692610b37d43a4aff572d6156b300b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:54.291500 7356 tablet_bootstrap.cc:492] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c: Bootstrap starting.
I20260807 02:21:54.292240 7358 tablet_bootstrap.cc:492] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a: Bootstrap starting.
I20260807 02:21:54.292426 7357 tablet_bootstrap.cc:492] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375: Bootstrap starting.
I20260807 02:21:54.292716 7356 tablet_bootstrap.cc:654] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:54.293315 7358 tablet_bootstrap.cc:654] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:54.293468 7357 tablet_bootstrap.cc:654] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:54.294176 7356 tablet_bootstrap.cc:492] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c: No bootstrap required, opened a new log
I20260807 02:21:54.294284 7356 ts_tablet_manager.cc:1397] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 02:21:54.294777 7356 raft_consensus.cc:348] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.294927 7356 raft_consensus.cc:374] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:54.294971 7356 raft_consensus.cc:729] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dcfdf2cb750443584931b0a7ab8ae4c, State: Initialized, Role: FOLLOWER
I20260807 02:21:54.295083 7357 tablet_bootstrap.cc:492] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375: No bootstrap required, opened a new log
I20260807 02:21:54.295190 7357 ts_tablet_manager.cc:1397] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 02:21:54.295128 7356 consensus_queue.cc:260] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.295454 7356 ts_tablet_manager.cc:1428] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 02:21:54.295738 7357 raft_consensus.cc:348] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.295871 7357 raft_consensus.cc:374] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:54.295925 7357 raft_consensus.cc:729] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a98433a0397243a8885c782f775af375, State: Initialized, Role: FOLLOWER
I20260807 02:21:54.296092 7357 consensus_queue.cc:260] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.296355 7358 tablet_bootstrap.cc:492] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a: No bootstrap required, opened a new log
I20260807 02:21:54.296450 7358 ts_tablet_manager.cc:1397] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260807 02:21:54.296608 7357 ts_tablet_manager.cc:1428] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260807 02:21:54.296933 7358 raft_consensus.cc:348] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.297060 7358 raft_consensus.cc:374] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:54.297091 7358 raft_consensus.cc:729] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012bac788de848698d856e91b895e20a, State: Initialized, Role: FOLLOWER
I20260807 02:21:54.297263 7358 consensus_queue.cc:260] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.297685 7358 ts_tablet_manager.cc:1428] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260807 02:21:54.375644 7307 raft_consensus.cc:482] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:21:54.375866 7307 raft_consensus.cc:504] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.376518 7307 leader_election.cc:288] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a98433a0397243a8885c782f775af375 (127.4.221.65:35129), 1dcfdf2cb750443584931b0a7ab8ae4c (127.4.221.66:39693)
I20260807 02:21:54.377203 6905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b692610b37d43a4aff572d6156b300b" candidate_uuid: "012bac788de848698d856e91b895e20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a98433a0397243a8885c782f775af375" is_pre_election: true
I20260807 02:21:54.377251 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b692610b37d43a4aff572d6156b300b" candidate_uuid: "012bac788de848698d856e91b895e20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" is_pre_election: true
I20260807 02:21:54.377482 6905 raft_consensus.cc:2454] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012bac788de848698d856e91b895e20a in term 0.
I20260807 02:21:54.377509 7052 raft_consensus.cc:2454] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 012bac788de848698d856e91b895e20a in term 0.
I20260807 02:21:54.378017 7123 leader_election.cc:302] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012bac788de848698d856e91b895e20a, a98433a0397243a8885c782f775af375; no voters:
I20260807 02:21:54.378245 7307 raft_consensus.cc:2781] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:21:54.378316 7307 raft_consensus.cc:482] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:21:54.378360 7307 raft_consensus.cc:3037] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:54.379683 7307 raft_consensus.cc:504] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.380070 7307 leader_election.cc:288] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [CANDIDATE]: Term 1 election: Requested vote from peers a98433a0397243a8885c782f775af375 (127.4.221.65:35129), 1dcfdf2cb750443584931b0a7ab8ae4c (127.4.221.66:39693)
I20260807 02:21:54.380558 6905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b692610b37d43a4aff572d6156b300b" candidate_uuid: "012bac788de848698d856e91b895e20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a98433a0397243a8885c782f775af375"
I20260807 02:21:54.380611 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b692610b37d43a4aff572d6156b300b" candidate_uuid: "012bac788de848698d856e91b895e20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c"
I20260807 02:21:54.380725 6905 raft_consensus.cc:3037] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:54.380750 7052 raft_consensus.cc:3037] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:54.382277 6905 raft_consensus.cc:2454] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012bac788de848698d856e91b895e20a in term 1.
I20260807 02:21:54.382277 7052 raft_consensus.cc:2454] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 012bac788de848698d856e91b895e20a in term 1.
I20260807 02:21:54.382614 7123 leader_election.cc:302] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 012bac788de848698d856e91b895e20a, a98433a0397243a8885c782f775af375; no voters:
I20260807 02:21:54.382834 7307 raft_consensus.cc:2781] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:21:54.383021 7307 raft_consensus.cc:686] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [term 1 LEADER]: Becoming Leader. State: Replica: 012bac788de848698d856e91b895e20a, State: Running, Role: LEADER
I20260807 02:21:54.383268 7307 consensus_queue.cc:237] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.385473 6745 catalog_manager.cc:5495] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a reported cstate change: term changed from 0 to 1, leader changed from <none> to 012bac788de848698d856e91b895e20a (127.4.221.67). New cstate: current_term: 1 leader_uuid: "012bac788de848698d856e91b895e20a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } health_report { overall_health: HEALTHY } } }
I20260807 02:21:54.405934 7365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:54.402112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46636 (local address 127.4.221.67:46219)
0807 02:21:54.402444 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:54.402453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:54.402466 (+ 13us) server_negotiation.cc:407] Connection header received
0807 02:21:54.402513 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:54.402515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:54.402565 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:54.402651 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:21:54.402870 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.403455 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.404532 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.404790 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.404922 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:54.405242 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:21:54.405345 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:21:54.405419 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:54.405643 (+ 224us) server_negotiation.cc:299] Negotiation successful
0807 02:21:54.405711 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":164,"threads_started":1}
I20260807 02:21:54.413882 7052 raft_consensus.cc:1261] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Refusing update from remote peer 012bac788de848698d856e91b895e20a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:54.413882 6905 raft_consensus.cc:1261] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Refusing update from remote peer 012bac788de848698d856e91b895e20a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:54.414356 7340 consensus_queue.cc:1035] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:21:54.414570 7307 consensus_queue.cc:1035] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:21:54.451551 6745 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:21:54.451763 6745 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:21:54.454123 6745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5: Sending DeleteTablet for 3 replicas of tablet 0b692610b37d43a4aff572d6156b300b
I20260807 02:21:54.454756 7174 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b692610b37d43a4aff572d6156b300b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:54 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:46620
I20260807 02:21:54.454764 6885 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b692610b37d43a4aff572d6156b300b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:54 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:52218
I20260807 02:21:54.454793 7030 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b692610b37d43a4aff572d6156b300b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:54 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:55504
I20260807 02:21:54.454993 7345 tablet_replica.cc:330] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c: stopping tablet replica
I20260807 02:21:54.454986 7346 tablet_replica.cc:330] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a: stopping tablet replica
I20260807 02:21:54.455160 7346 raft_consensus.cc:2229] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:21:54.455168 7345 raft_consensus.cc:2229] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:21:54.455322 7345 raft_consensus.cc:2258] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:54.455394 7346 raft_consensus.cc:2258] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:54.455614 7347 tablet_replica.cc:330] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375: stopping tablet replica
I20260807 02:21:54.455783 7347 raft_consensus.cc:2229] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:21:54.455924 7347 raft_consensus.cc:2258] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:54.456312 7345 ts_tablet_manager.cc:1905] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:21:54.456326 6745 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33930:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 02:21:54.456593 7346 ts_tablet_manager.cc:1905] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:21:54.456738 7347 ts_tablet_manager.cc:1905] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 02:21:54.456847 6745 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:21:54.459478 7347 ts_tablet_manager.cc:1918] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:21:54.459544 7347 log.cc:1196] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-0/wal/wals/0b692610b37d43a4aff572d6156b300b
I20260807 02:21:54.459619 7346 ts_tablet_manager.cc:1918] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:21:54.459619 7345 ts_tablet_manager.cc:1918] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:21:54.459699 7345 log.cc:1196] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/0b692610b37d43a4aff572d6156b300b
I20260807 02:21:54.459699 7346 log.cc:1196] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/0b692610b37d43a4aff572d6156b300b
I20260807 02:21:54.459748 7347 ts_tablet_manager.cc:1939] T 0b692610b37d43a4aff572d6156b300b P a98433a0397243a8885c782f775af375: Deleting consensus metadata
I20260807 02:21:54.459985 7345 ts_tablet_manager.cc:1939] T 0b692610b37d43a4aff572d6156b300b P 1dcfdf2cb750443584931b0a7ab8ae4c: Deleting consensus metadata
I20260807 02:21:54.460016 7346 ts_tablet_manager.cc:1939] T 0b692610b37d43a4aff572d6156b300b P 012bac788de848698d856e91b895e20a: Deleting consensus metadata
I20260807 02:21:54.460682 6730 catalog_manager.cc:4841] TS a98433a0397243a8885c782f775af375 (127.4.221.65:35129): tablet 0b692610b37d43a4aff572d6156b300b (table test-table [id=66709c1e13a3421c9c145096a115379f]) successfully deleted
I20260807 02:21:54.461076 6730 catalog_manager.cc:4841] TS 1dcfdf2cb750443584931b0a7ab8ae4c (127.4.221.66:39693): tablet 0b692610b37d43a4aff572d6156b300b (table test-table [id=66709c1e13a3421c9c145096a115379f]) successfully deleted
I20260807 02:21:54.461606 6731 catalog_manager.cc:4841] TS 012bac788de848698d856e91b895e20a (127.4.221.67:46219): tablet 0b692610b37d43a4aff572d6156b300b (table test-table [id=66709c1e13a3421c9c145096a115379f]) successfully deleted
I20260807 02:21:54.464236 6885 tablet_service.cc:1467] Processing CreateTablet for tablet 5662d96286e44f269bf95406e599df71 (DEFAULT_TABLE table=test-table [id=796b4c87124a44d08cd6b6258c0b0be1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:54.464280 7030 tablet_service.cc:1467] Processing CreateTablet for tablet 5662d96286e44f269bf95406e599df71 (DEFAULT_TABLE table=test-table [id=796b4c87124a44d08cd6b6258c0b0be1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:54.464340 7174 tablet_service.cc:1467] Processing CreateTablet for tablet 5662d96286e44f269bf95406e599df71 (DEFAULT_TABLE table=test-table [id=796b4c87124a44d08cd6b6258c0b0be1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:21:54.464527 6885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5662d96286e44f269bf95406e599df71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:54.464576 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5662d96286e44f269bf95406e599df71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:54.464576 7174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5662d96286e44f269bf95406e599df71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:54.466770 7357 tablet_bootstrap.cc:492] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375: Bootstrap starting.
I20260807 02:21:54.467046 7356 tablet_bootstrap.cc:492] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c: Bootstrap starting.
I20260807 02:21:54.467188 7358 tablet_bootstrap.cc:492] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a: Bootstrap starting.
I20260807 02:21:54.467927 7357 tablet_bootstrap.cc:654] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:54.467950 7356 tablet_bootstrap.cc:654] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:54.467995 7358 tablet_bootstrap.cc:654] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:54.469250 7358 tablet_bootstrap.cc:492] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a: No bootstrap required, opened a new log
I20260807 02:21:54.469309 7357 tablet_bootstrap.cc:492] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375: No bootstrap required, opened a new log
I20260807 02:21:54.469363 7358 ts_tablet_manager.cc:1397] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260807 02:21:54.469400 7357 ts_tablet_manager.cc:1397] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 02:21:54.469544 7356 tablet_bootstrap.cc:492] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c: No bootstrap required, opened a new log
I20260807 02:21:54.469640 7356 ts_tablet_manager.cc:1397] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 02:21:54.469902 7358 raft_consensus.cc:348] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.469945 7357 raft_consensus.cc:348] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.470048 7358 raft_consensus.cc:374] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:54.470062 7357 raft_consensus.cc:374] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:54.470088 7358 raft_consensus.cc:729] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 012bac788de848698d856e91b895e20a, State: Initialized, Role: FOLLOWER
I20260807 02:21:54.470096 7357 raft_consensus.cc:729] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a98433a0397243a8885c782f775af375, State: Initialized, Role: FOLLOWER
I20260807 02:21:54.470223 7358 consensus_queue.cc:260] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.470270 7357 consensus_queue.cc:260] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.470284 7356 raft_consensus.cc:348] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.470392 7356 raft_consensus.cc:374] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:54.470436 7356 raft_consensus.cc:729] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dcfdf2cb750443584931b0a7ab8ae4c, State: Initialized, Role: FOLLOWER
I20260807 02:21:54.470557 7358 ts_tablet_manager.cc:1428] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260807 02:21:54.470604 7357 ts_tablet_manager.cc:1428] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260807 02:21:54.470558 7356 consensus_queue.cc:260] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.470816 7356 ts_tablet_manager.cc:1428] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 02:21:54.523939 7362 raft_consensus.cc:482] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:21:54.524173 7362 raft_consensus.cc:504] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.525530 7362 leader_election.cc:288] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dcfdf2cb750443584931b0a7ab8ae4c (127.4.221.66:39693), 012bac788de848698d856e91b895e20a (127.4.221.67:46219)
I20260807 02:21:54.530331 7372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:54.525902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:39693 (local address 127.4.221.65:33453)
0807 02:21:54.526400 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:54.526415 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:54.526598 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:54.526938 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:54.526941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:54.526958 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:54.527137 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:54.527142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.528044 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.528047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:54.529013 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:54.529022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.529778 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.529782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:54.529911 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:54.529932 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 02:21:54.529996 (+ 64us) client_negotiation.cc:240] Negotiation successful
0807 02:21:54.530069 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":265,"spinlock_wait_cycles":18304,"thread_start_us":113,"threads_started":1}
I20260807 02:21:54.530345 7373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:54.526055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:33453 (local address 127.4.221.66:39693)
0807 02:21:54.526408 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:54.526430 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:54.526544 (+ 114us) server_negotiation.cc:407] Connection header received
0807 02:21:54.526739 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:54.526742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:54.526797 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:54.526908 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:54.527254 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.527874 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.529143 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.529623 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.529779 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:54.529865 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:54.530044 (+ 179us) server_negotiation.cc:299] Negotiation successful
0807 02:21:54.530152 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":119,"threads_started":1}
I20260807 02:21:54.530675 7365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:54.526038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:43861 (local address 127.4.221.67:46219)
0807 02:21:54.526221 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:54.526228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:54.526486 (+ 258us) server_negotiation.cc:407] Connection header received
0807 02:21:54.526733 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:54.526743 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:54.526797 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:54.526861 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:21:54.527296 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.527849 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.529181 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.529620 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.529763 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:54.529851 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:54.530339 (+ 488us) server_negotiation.cc:299] Negotiation successful
0807 02:21:54.530451 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 02:21:54.531015 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5662d96286e44f269bf95406e599df71" candidate_uuid: "a98433a0397243a8885c782f775af375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" is_pre_election: true
I20260807 02:21:54.531144 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:54.525902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:46219 (local address 127.4.221.65:43861)
0807 02:21:54.526394 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:54.526409 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:54.526596 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:54.526904 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:54.526908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:54.526924 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:21:54.527149 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:54.527155 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.527976 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.527980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:54.529025 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:54.529035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.529761 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.529765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:54.530171 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:54.530196 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 02:21:54.530868 (+ 672us) client_negotiation.cc:240] Negotiation successful
0807 02:21:54.530961 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":83,"spinlock_wait_cycles":37248,"thread_start_us":125,"threads_started":1}
I20260807 02:21:54.531231 7052 raft_consensus.cc:2454] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a98433a0397243a8885c782f775af375 in term 0.
I20260807 02:21:54.531704 7194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5662d96286e44f269bf95406e599df71" candidate_uuid: "a98433a0397243a8885c782f775af375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012bac788de848698d856e91b895e20a" is_pre_election: true
I20260807 02:21:54.531729 6836 leader_election.cc:302] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dcfdf2cb750443584931b0a7ab8ae4c, a98433a0397243a8885c782f775af375; no voters:
I20260807 02:21:54.531898 7194 raft_consensus.cc:2454] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a98433a0397243a8885c782f775af375 in term 0.
I20260807 02:21:54.532099 7362 raft_consensus.cc:2781] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:21:54.532177 7362 raft_consensus.cc:482] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:21:54.532233 7362 raft_consensus.cc:3037] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:54.534214 7362 raft_consensus.cc:504] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.534626 7362 leader_election.cc:288] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [CANDIDATE]: Term 1 election: Requested vote from peers 1dcfdf2cb750443584931b0a7ab8ae4c (127.4.221.66:39693), 012bac788de848698d856e91b895e20a (127.4.221.67:46219)
I20260807 02:21:54.535043 7052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5662d96286e44f269bf95406e599df71" candidate_uuid: "a98433a0397243a8885c782f775af375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c"
I20260807 02:21:54.535171 7052 raft_consensus.cc:3037] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:54.535183 7194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5662d96286e44f269bf95406e599df71" candidate_uuid: "a98433a0397243a8885c782f775af375" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "012bac788de848698d856e91b895e20a"
I20260807 02:21:54.535513 7194 raft_consensus.cc:3037] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:54.536423 7052 raft_consensus.cc:2454] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98433a0397243a8885c782f775af375 in term 1.
I20260807 02:21:54.536751 6836 leader_election.cc:302] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dcfdf2cb750443584931b0a7ab8ae4c, a98433a0397243a8885c782f775af375; no voters:
I20260807 02:21:54.536954 7362 raft_consensus.cc:2781] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:21:54.537204 7194 raft_consensus.cc:2454] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a98433a0397243a8885c782f775af375 in term 1.
I20260807 02:21:54.537238 7362 raft_consensus.cc:686] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [term 1 LEADER]: Becoming Leader. State: Replica: a98433a0397243a8885c782f775af375, State: Running, Role: LEADER
I20260807 02:21:54.537765 7362 consensus_queue.cc:237] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } }
I20260807 02:21:54.540789 6746 catalog_manager.cc:5495] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 reported cstate change: term changed from 0 to 1, leader changed from <none> to a98433a0397243a8885c782f775af375 (127.4.221.65). New cstate: current_term: 1 leader_uuid: "a98433a0397243a8885c782f775af375" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a98433a0397243a8885c782f775af375" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 35129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 } health_report { overall_health: UNKNOWN } } }
I20260807 02:21:54.560950 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:54.556724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.4.221.65:35129)
0807 02:21:54.557092 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:54.557099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:54.557111 (+ 12us) server_negotiation.cc:407] Connection header received
0807 02:21:54.557197 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:54.557200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:54.557261 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:54.557429 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:21:54.557653 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.558436 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.559361 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.559644 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.559726 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:54.559967 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:21:54.560590 (+ 623us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:21:54.560687 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:54.560745 (+ 58us) server_negotiation.cc:299] Negotiation successful
0807 02:21:54.560798 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":176,"threads_started":1}
I20260807 02:21:54.572535 7373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:54.569349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:44209 (local address 127.4.221.66:39693)
0807 02:21:54.569494 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:54.569498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:54.569695 (+ 197us) server_negotiation.cc:407] Connection header received
0807 02:21:54.569799 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:54.569802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:54.569883 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:54.569947 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:21:54.570209 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.570731 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.571586 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.571796 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.571874 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:54.572140 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:21:54.572216 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:21:54.572270 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:54.572352 (+ 82us) server_negotiation.cc:299] Negotiation successful
0807 02:21:54.572403 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 02:21:54.572592 7380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:54.569228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:39693 (local address 127.4.221.65:44209)
0807 02:21:54.569600 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:54.569614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:54.569710 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:54.569977 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:54.569979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:54.569986 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:21:54.570100 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:54.570104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.570828 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.570831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:54.571481 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:54.571490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.571895 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.571897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:54.571972 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:54.572016 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:21:54.572291 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:21:54.572297 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 02:21:54.572345 (+ 48us) client_negotiation.cc:240] Negotiation successful
0807 02:21:54.572408 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":154,"threads_started":1}
I20260807 02:21:54.579025 7052 raft_consensus.cc:1261] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Refusing update from remote peer a98433a0397243a8885c782f775af375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:54.579025 7194 raft_consensus.cc:1261] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a [term 1 FOLLOWER]: Refusing update from remote peer a98433a0397243a8885c782f775af375: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:21:54.579548 7362 consensus_queue.cc:1035] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "012bac788de848698d856e91b895e20a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 46219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:21:54.579773 7375 consensus_queue.cc:1035] T 5662d96286e44f269bf95406e599df71 P a98433a0397243a8885c782f775af375 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dcfdf2cb750443584931b0a7ab8ae4c" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:21:54.599771 7385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:54.595440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:35129 (local address 127.4.221.66:41437)
0807 02:21:54.595771 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:54.595786 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:54.595923 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:54.596185 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:54.596187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:54.596194 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:21:54.596309 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:54.596313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.597382 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.597385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:54.598108 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:54.598129 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.598751 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.598756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:54.598903 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:54.598941 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:21:54.599382 (+ 441us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:21:54.599390 (+ 8us) client_negotiation.cc:769] Sending connection context
0807 02:21:54.599455 (+ 65us) client_negotiation.cc:240] Negotiation successful
0807 02:21:54.599554 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":142,"threads_started":1}
I20260807 02:21:54.599768 7377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:54.595522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:41437 (local address 127.4.221.65:35129)
0807 02:21:54.595679 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:54.595687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:54.595892 (+ 205us) server_negotiation.cc:407] Connection header received
0807 02:21:54.596031 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:54.596034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:54.596095 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:54.596182 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:21:54.596437 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.597259 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.598225 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:54.598459 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:54.598553 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:54.599139 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:21:54.599264 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:21:54.599362 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:54.599497 (+ 135us) server_negotiation.cc:299] Negotiation successful
0807 02:21:54.599576 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260807 02:21:54.600708 7326 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260807 02:21:56.170585 6746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:21:56.170795 6746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33930:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:21:56.173619 6746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d66968a8cb1c4e70aadc535b7a9e33b5: Sending DeleteTablet for 3 replicas of tablet 5662d96286e44f269bf95406e599df71
I20260807 02:21:56.174329 6885 tablet_service.cc:1514] Processing DeleteTablet for tablet 5662d96286e44f269bf95406e599df71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:56 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:52218
I20260807 02:21:56.174474 7030 tablet_service.cc:1514] Processing DeleteTablet for tablet 5662d96286e44f269bf95406e599df71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:56 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:55504
I20260807 02:21:56.174734 7174 tablet_service.cc:1514] Processing DeleteTablet for tablet 5662d96286e44f269bf95406e599df71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:21:56 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:46620
I20260807 02:21:56.174892 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 6820
I20260807 02:21:56.174933 7410 tablet_replica.cc:330] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c: stopping tablet replica
I20260807 02:21:56.177104 7410 raft_consensus.cc:2229] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:21:56.177274 7410 raft_consensus.cc:2258] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:56.178136 7411 tablet_replica.cc:330] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a: stopping tablet replica
I20260807 02:21:56.178188 7410 ts_tablet_manager.cc:1905] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:21:56.178330 7411 raft_consensus.cc:2229] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:21:56.178471 7411 raft_consensus.cc:2258] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:21:56.179292 7411 ts_tablet_manager.cc:1905] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:21:56.180977 7410 ts_tablet_manager.cc:1918] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 02:21:56.181083 7410 log.cc:1196] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/5662d96286e44f269bf95406e599df71
I20260807 02:21:56.181391 7410 ts_tablet_manager.cc:1939] T 5662d96286e44f269bf95406e599df71 P 1dcfdf2cb750443584931b0a7ab8ae4c: Deleting consensus metadata
I20260807 02:21:56.182300 6730 catalog_manager.cc:4841] TS 1dcfdf2cb750443584931b0a7ab8ae4c (127.4.221.66:39693): tablet 5662d96286e44f269bf95406e599df71 (table test-table [id=796b4c87124a44d08cd6b6258c0b0be1]) successfully deleted
I20260807 02:21:56.183169 7411 ts_tablet_manager.cc:1918] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260807 02:21:56.183259 7411 log.cc:1196] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/5662d96286e44f269bf95406e599df71
I20260807 02:21:56.183537 7411 ts_tablet_manager.cc:1939] T 5662d96286e44f269bf95406e599df71 P 012bac788de848698d856e91b895e20a: Deleting consensus metadata
I20260807 02:21:56.184612 6731 catalog_manager.cc:4841] TS 012bac788de848698d856e91b895e20a (127.4.221.67:46219): tablet 5662d96286e44f269bf95406e599df71 (table test-table [id=796b4c87124a44d08cd6b6258c0b0be1]) successfully deleted
W20260807 02:21:56.186858 6730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:35129 (error 108)
I20260807 02:21:56.187209 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 6964
I20260807 02:21:56.191933 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:56.189966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:35129 (local address 127.0.0.1:52228)
0807 02:21:56.191657 (+ 1691us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:56.191767 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:35129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1564,"thread_start_us":115,"threads_started":1}
W20260807 02:21:56.192122 6730 catalog_manager.cc:4584] TS a98433a0397243a8885c782f775af375 (127.4.221.65:35129): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5662d96286e44f269bf95406e599df71: Network error: Client connection negotiation failed: client connection to 127.4.221.65:35129: connect: Connection refused (error 111)
I20260807 02:21:56.197118 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 7107
I20260807 02:21:56.206264 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 6714
2026-08-07T02:21:56Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6498 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:21:56 dist-test-slave-69rg krb5kdc[7416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:21:56 dist-test-slave-69rg krb5kdc[7416](info): set up 2 sockets
Aug 07 02:21:56 dist-test-slave-69rg krb5kdc[7416](info): commencing operation
krb5kdc: starting...
W20260807 02:21:57.286033 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:21:57 dist-test-slave-69rg krb5kdc[7416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:21:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:21:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:21:57.426007 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44673
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:44673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:57.543054 7430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:57.543390 7430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:57.543481 7430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:57.547670 7430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:21:57.547780 7430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:57.547818 7430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:57.547875 7430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:21:57.547931 7430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:21:57.553553 7430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:44673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:57.554838 7430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:57.555954 7430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:57.565433 7435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:57.565512 7430 server_base.cc:1034] running on GCE node
W20260807 02:21:57.565457 7436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:57.565448 7438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:57.566286 7430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:57.567524 7430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:57.568729 7430 hybrid_clock.cc:648] HybridClock initialized: now 1786069317568703 us; error 45 us; skew 500 ppm
Aug 07 02:21:57 dist-test-slave-69rg krb5kdc[7416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:57.571810 7430 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:21:57.572849 7430 webserver.cc:460] Webserver started at http://127.4.221.126:44423/ using document root <none> and password file <none>
I20260807 02:21:57.573442 7430 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:57.573537 7430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:57.573794 7430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:57.576380 7430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "18ad9cabfb3544d48cc3723fe850b30c"
format_stamp: "Formatted at 2026-08-07 02:21:57 on dist-test-slave-69rg"
server_key: "5918d3492064164cfa13ffd31b544a7f"
server_key_iv: "5676bcf7df41adfb5446e76c0cfe3543"
server_key_version: "encryptionkey@0"
I20260807 02:21:57.577008 7430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "18ad9cabfb3544d48cc3723fe850b30c"
format_stamp: "Formatted at 2026-08-07 02:21:57 on dist-test-slave-69rg"
server_key: "5918d3492064164cfa13ffd31b544a7f"
server_key_iv: "5676bcf7df41adfb5446e76c0cfe3543"
server_key_version: "encryptionkey@0"
I20260807 02:21:57.581015 7430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 02:21:57.583623 7445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:57.584959 7430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:21:57.585065 7430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "18ad9cabfb3544d48cc3723fe850b30c"
format_stamp: "Formatted at 2026-08-07 02:21:57 on dist-test-slave-69rg"
server_key: "5918d3492064164cfa13ffd31b544a7f"
server_key_iv: "5676bcf7df41adfb5446e76c0cfe3543"
server_key_version: "encryptionkey@0"
I20260807 02:21:57.585163 7430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:57.603788 7430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:57.607005 7430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:57.607190 7430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:57.614880 7430 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:44673
I20260807 02:21:57.614908 7500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:44673 every 8 connection(s)
I20260807 02:21:57.615931 7430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:21:57.618898 7501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:57.624320 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 7430
I20260807 02:21:57.624433 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:21:57.624742 4981 external_mini_cluster.cc:1428] Setting key 7332f9630a4e3c66d039d5f9317e6055
I20260807 02:21:57.625150 7501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c: Bootstrap starting.
I20260807 02:21:57.628240 7501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:57.629268 7501 log.cc:824] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c: Log is configured to *not* fsync() on all Append() calls
I20260807 02:21:57.631918 7501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c: No bootstrap required, opened a new log
Aug 07 02:21:57 dist-test-slave-69rg krb5kdc[7416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:57.636448 7501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ad9cabfb3544d48cc3723fe850b30c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44673 } }
I20260807 02:21:57.636813 7501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:57.636922 7501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18ad9cabfb3544d48cc3723fe850b30c, State: Initialized, Role: FOLLOWER
I20260807 02:21:57.637504 7501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ad9cabfb3544d48cc3723fe850b30c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44673 } }
I20260807 02:21:57.637646 7501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:21:57.637740 7501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:21:57.637872 7501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:57.638934 7501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ad9cabfb3544d48cc3723fe850b30c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44673 } }
I20260807 02:21:57.639340 7501 leader_election.cc:302] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18ad9cabfb3544d48cc3723fe850b30c; no voters:
I20260807 02:21:57.639745 7501 leader_election.cc:288] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:21:57.639833 7507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:21:57.640065 7507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [term 1 LEADER]: Becoming Leader. State: Replica: 18ad9cabfb3544d48cc3723fe850b30c, State: Running, Role: LEADER
I20260807 02:21:57.640448 7507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ad9cabfb3544d48cc3723fe850b30c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44673 } }
I20260807 02:21:57.641201 7501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:21:57.642578 7509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18ad9cabfb3544d48cc3723fe850b30c. Latest consensus state: current_term: 1 leader_uuid: "18ad9cabfb3544d48cc3723fe850b30c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ad9cabfb3544d48cc3723fe850b30c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44673 } } }
I20260807 02:21:57.642580 7508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18ad9cabfb3544d48cc3723fe850b30c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18ad9cabfb3544d48cc3723fe850b30c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44673 } } }
I20260807 02:21:57.642720 7509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [sys.catalog]: This master's current role is: LEADER
I20260807 02:21:57.642745 7508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c [sys.catalog]: This master's current role is: LEADER
I20260807 02:21:57.642906 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:57.626477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43638 (local address 127.4.221.126:44673)
0807 02:21:57.626966 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:57.626987 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:57.627033 (+ 46us) server_negotiation.cc:407] Connection header received
0807 02:21:57.627928 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:57.627954 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:57.628431 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:57.628854 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:57.629754 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:57.631195 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:57.632171 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:57.632714 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:57.632892 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:57.636129 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:57.636135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:57.636175 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:57.636218 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:57.638397 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:57.638898 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:57.638905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:57.638911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:57.639106 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:57.639418 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:57.639422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:57.639424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:57.640023 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:57.640912 (+ 889us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:57.641041 (+ 129us) server_negotiation.cc:299] Negotiation successful
0807 02:21:57.641355 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":149,"threads_started":1}
I20260807 02:21:57.643180 7516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:21:57.646962 7516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:21:57.653062 7516 catalog_manager.cc:1347] Generated new cluster ID: 6cbb8c75bb0e4181942e547d45c48a58
I20260807 02:21:57.653163 7516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:21:57.663472 7516 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:21:57.664901 7516 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:21:57.673110 7516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18ad9cabfb3544d48cc3723fe850b30c: Generated new TSK 0
I20260807 02:21:57.673954 7516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:21:57.713753 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44673
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:57.849843 7528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:57.850163 7528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:57.850212 7528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:57.854548 7528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:57.854632 7528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:57.854696 7528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:21:57.860394 7528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:57.861601 7528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:57.862775 7528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 02:21:57.873360 7528 server_base.cc:1034] running on GCE node
W20260807 02:21:57.873287 7536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:57.873380 7534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:57.873298 7533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:57.874042 7528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:57.874724 7528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:57.875909 7528 hybrid_clock.cc:648] HybridClock initialized: now 1786069317875878 us; error 43 us; skew 500 ppm
Aug 07 02:21:57 dist-test-slave-69rg krb5kdc[7416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:57.879030 7528 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:21:57.880010 7528 webserver.cc:460] Webserver started at http://127.4.221.65:38079/ using document root <none> and password file <none>
I20260807 02:21:57.880620 7528 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:57.880723 7528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:57.880968 7528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:57.883329 7528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "4623a415fd944462bdcdf69be6706387"
format_stamp: "Formatted at 2026-08-07 02:21:57 on dist-test-slave-69rg"
server_key: "afbbdd1f34628fa0aaba70b1de21cb6a"
server_key_iv: "e9c8cbec5d87b8edf2b1269c9dfad852"
server_key_version: "encryptionkey@0"
I20260807 02:21:57.883950 7528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "4623a415fd944462bdcdf69be6706387"
format_stamp: "Formatted at 2026-08-07 02:21:57 on dist-test-slave-69rg"
server_key: "afbbdd1f34628fa0aaba70b1de21cb6a"
server_key_iv: "e9c8cbec5d87b8edf2b1269c9dfad852"
server_key_version: "encryptionkey@0"
I20260807 02:21:57.888067 7528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:21:57.891153 7543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:57.892584 7528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:21:57.892760 7528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "4623a415fd944462bdcdf69be6706387"
format_stamp: "Formatted at 2026-08-07 02:21:57 on dist-test-slave-69rg"
server_key: "afbbdd1f34628fa0aaba70b1de21cb6a"
server_key_iv: "e9c8cbec5d87b8edf2b1269c9dfad852"
server_key_version: "encryptionkey@0"
I20260807 02:21:57.892902 7528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:57.915437 7528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:57.918845 7528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:57.919088 7528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:57.919771 7528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:21:57.920727 7528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:21:57.920807 7528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:57.920898 7528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:21:57.920955 7528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:57.931362 7528 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:36591
I20260807 02:21:57.931389 7660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:36591 every 8 connection(s)
I20260807 02:21:57.932382 7528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:21:57.933116 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 7528
I20260807 02:21:57.933247 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:21:57.933537 4981 external_mini_cluster.cc:1428] Setting key 8591f7351e48a58a80905a9bf40be140
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
Aug 07 02:21:57 dist-test-slave-69rg krb5kdc[7416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069317, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
I20260807 02:21:57.951761 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:57.934572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:35015 (local address 127.4.221.126:44673)
0807 02:21:57.934735 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:57.934741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:57.938042 (+ 3301us) server_negotiation.cc:407] Connection header received
0807 02:21:57.939571 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:57.939575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:57.939638 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:57.939746 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:57.940995 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:57.942119 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:57.943520 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:57.944009 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:57.944139 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:57.946963 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:57.946966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:57.946968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:57.947000 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:57.948922 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:57.949724 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:57.949732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:57.949735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:57.949879 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:57.950250 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:57.950256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:57.950261 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:57.950575 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:57.950711 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:57.951250 (+ 539us) server_negotiation.cc:299] Negotiation successful
0807 02:21:57.951430 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 02:21:57.952441 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:57.935046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44673 (local address 127.4.221.65:35015)
0807 02:21:57.937806 (+ 2760us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:57.937929 (+ 123us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:57.939332 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:57.939928 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:57.939936 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:57.940321 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:57.940817 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:57.940832 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:57.942336 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:57.942339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:57.943374 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:57.943390 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:57.944237 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:57.944240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:57.944401 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:57.945119 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:57.945137 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:57.946775 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:57.949091 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:57.949097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:57.949114 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:57.949579 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:57.949981 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:57.949983 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:57.949985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:57.950137 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:57.950702 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:57.950708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:57.950990 (+ 282us) client_negotiation.cc:769] Sending connection context
0807 02:21:57.951159 (+ 169us) client_negotiation.cc:240] Negotiation successful
0807 02:21:57.951452 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2537,"thread_start_us":136,"threads_started":1}
I20260807 02:21:57.953848 7661 heartbeater.cc:347] Connected to a master server at 127.4.221.126:44673
I20260807 02:21:57.954123 7661 heartbeater.cc:464] Registering TS with master...
I20260807 02:21:57.954658 7661 heartbeater.cc:511] Master 127.4.221.126:44673 requested a full tablet report, sending...
I20260807 02:21:57.956405 7464 ts_manager.cc:194] Registered new tserver with Master: 4623a415fd944462bdcdf69be6706387 (127.4.221.65:36591)
I20260807 02:21:57.957603 7464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:35015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:21:57.969283 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44673
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:58.100225 7666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:58.100566 7666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:58.100649 7666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:58.104993 7666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:58.105098 7666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:58.105206 7666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:21:58.111040 7666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:58.112252 7666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:58.113359 7666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:58.124289 7674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:58.124397 7672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:58.124302 7671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:58.124516 7666 server_base.cc:1034] running on GCE node
I20260807 02:21:58.125082 7666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:58.125984 7666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:58.127166 7666 hybrid_clock.cc:648] HybridClock initialized: now 1786069318127143 us; error 37 us; skew 500 ppm
Aug 07 02:21:58 dist-test-slave-69rg krb5kdc[7416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069318, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:58.130587 7666 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:21:58.131553 7666 webserver.cc:460] Webserver started at http://127.4.221.66:36525/ using document root <none> and password file <none>
I20260807 02:21:58.132181 7666 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:58.132249 7666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:58.132480 7666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:58.134733 7666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "0bca0a4be09d440d87a6177979417a45"
format_stamp: "Formatted at 2026-08-07 02:21:58 on dist-test-slave-69rg"
server_key: "983380f5454c29f8cae89b55c7aa1fc2"
server_key_iv: "53ad179a06bd9f520ca4c48e5270cbcf"
server_key_version: "encryptionkey@0"
I20260807 02:21:58.135344 7666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "0bca0a4be09d440d87a6177979417a45"
format_stamp: "Formatted at 2026-08-07 02:21:58 on dist-test-slave-69rg"
server_key: "983380f5454c29f8cae89b55c7aa1fc2"
server_key_iv: "53ad179a06bd9f520ca4c48e5270cbcf"
server_key_version: "encryptionkey@0"
I20260807 02:21:58.139350 7666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:21:58.142007 7681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:58.143512 7666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:21:58.143669 7666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "0bca0a4be09d440d87a6177979417a45"
format_stamp: "Formatted at 2026-08-07 02:21:58 on dist-test-slave-69rg"
server_key: "983380f5454c29f8cae89b55c7aa1fc2"
server_key_iv: "53ad179a06bd9f520ca4c48e5270cbcf"
server_key_version: "encryptionkey@0"
I20260807 02:21:58.143788 7666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:58.155059 7666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:58.158519 7666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:58.158751 7666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:58.159411 7666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:21:58.160316 7666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:21:58.160367 7666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:58.160439 7666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:21:58.160478 7666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:58.170257 7666 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:43073
I20260807 02:21:58.170269 7798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:43073 every 8 connection(s)
I20260807 02:21:58.171221 7666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:21:58.177606 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 7666
I20260807 02:21:58.177735 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:21:58.178071 4981 external_mini_cluster.cc:1428] Setting key b219aadf6f6603d2e0c2b17fed8035e8
Aug 07 02:21:58 dist-test-slave-69rg krb5kdc[7416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069318, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:58.186604 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:58.173076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:37719 (local address 127.4.221.126:44673)
0807 02:21:58.173237 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:58.173242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:58.174241 (+ 999us) server_negotiation.cc:407] Connection header received
0807 02:21:58.175676 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:58.175679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:58.175725 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:58.175797 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:58.176915 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:58.177453 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:58.179096 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:58.179427 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:58.179565 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:58.182528 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:58.182533 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:58.182535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:58.182571 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:58.184175 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:58.184795 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:58.184798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:58.184799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:58.184883 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:58.185265 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:58.185269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:58.185272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:58.185479 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:58.185593 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:58.186266 (+ 673us) server_negotiation.cc:299] Negotiation successful
0807 02:21:58.186418 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:21:58.187511 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:58.173542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44673 (local address 127.4.221.66:37719)
0807 02:21:58.174091 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:58.174135 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:58.175478 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:58.175926 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:58.175932 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:58.176307 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:58.176782 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:58.176798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:58.177592 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:58.177598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:58.178966 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:58.178976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:58.179609 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:58.179613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:58.179796 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:58.180454 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:58.180471 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:58.182333 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:58.184322 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:58.184329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:58.184347 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:58.184663 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:58.184985 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:58.184989 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:58.184991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:58.185156 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:58.185608 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:58.185624 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:58.185995 (+ 371us) client_negotiation.cc:769] Sending connection context
0807 02:21:58.186203 (+ 208us) client_negotiation.cc:240] Negotiation successful
0807 02:21:58.186576 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
I20260807 02:21:58.188855 7799 heartbeater.cc:347] Connected to a master server at 127.4.221.126:44673
I20260807 02:21:58.189242 7799 heartbeater.cc:464] Registering TS with master...
I20260807 02:21:58.189937 7799 heartbeater.cc:511] Master 127.4.221.126:44673 requested a full tablet report, sending...
I20260807 02:21:58.191056 7464 ts_manager.cc:194] Registered new tserver with Master: 0bca0a4be09d440d87a6177979417a45 (127.4.221.66:43073)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
I20260807 02:21:58.191905 7464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:37719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:21:58.210044 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44673
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:42589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:58.329397 7804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:58.329708 7804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:58.329754 7804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:58.334035 7804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:58.334115 7804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:58.334183 7804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:21:58.340093 7804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:42589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:58.341626 7804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:58.342928 7804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:21:58.353389 7809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:58.353329 7812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:58.353333 7810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:58.353628 7804 server_base.cc:1034] running on GCE node
I20260807 02:21:58.354127 7804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:58.354809 7804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:58.355997 7804 hybrid_clock.cc:648] HybridClock initialized: now 1786069318355983 us; error 28 us; skew 500 ppm
Aug 07 02:21:58 dist-test-slave-69rg krb5kdc[7416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069318, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:58.358985 7804 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:21:58.359901 7804 webserver.cc:460] Webserver started at http://127.4.221.67:43999/ using document root <none> and password file <none>
I20260807 02:21:58.360504 7804 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:58.360569 7804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:58.360826 7804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:58.363173 7804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "3c14e5e261a44b539a443af50e31867b"
format_stamp: "Formatted at 2026-08-07 02:21:58 on dist-test-slave-69rg"
server_key: "1df1d99a22e6d44bd2641d6105e48c42"
server_key_iv: "4d47c65c151b8a96c16d968d5b973847"
server_key_version: "encryptionkey@0"
I20260807 02:21:58.363796 7804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "3c14e5e261a44b539a443af50e31867b"
format_stamp: "Formatted at 2026-08-07 02:21:58 on dist-test-slave-69rg"
server_key: "1df1d99a22e6d44bd2641d6105e48c42"
server_key_iv: "4d47c65c151b8a96c16d968d5b973847"
server_key_version: "encryptionkey@0"
I20260807 02:21:58.367861 7804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:21:58.370447 7820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:58.371727 7804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:21:58.371887 7804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "3c14e5e261a44b539a443af50e31867b"
format_stamp: "Formatted at 2026-08-07 02:21:58 on dist-test-slave-69rg"
server_key: "1df1d99a22e6d44bd2641d6105e48c42"
server_key_iv: "4d47c65c151b8a96c16d968d5b973847"
server_key_version: "encryptionkey@0"
I20260807 02:21:58.372013 7804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:58.392292 7804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:58.395824 7804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:58.396050 7804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:58.396703 7804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:21:58.397688 7804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:21:58.397763 7804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:58.397861 7804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:21:58.397915 7804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:58.407612 7804 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:33861
I20260807 02:21:58.407660 7935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:33861 every 8 connection(s)
I20260807 02:21:58.408584 7804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:21:58.418458 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 7804
Aug 07 02:21:58 dist-test-slave-69rg krb5kdc[7416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069318, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:58.418592 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:21:58.418885 4981 external_mini_cluster.cc:1428] Setting key 37dbf3b008ccfe61f84e374b2fcea668
I20260807 02:21:58.423091 7505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:58.410517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:37323 (local address 127.4.221.126:44673)
0807 02:21:58.410669 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:58.410673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:58.411522 (+ 849us) server_negotiation.cc:407] Connection header received
0807 02:21:58.412900 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:58.412904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:58.412965 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:58.413074 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:58.414345 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:58.414890 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:58.416049 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:58.416278 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:58.416390 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:58.419252 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:58.419255 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:58.419257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:58.419283 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:58.420795 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:58.421432 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:58.421436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:58.421438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:58.421524 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:58.421945 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:58.421948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:58.421950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:58.422148 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:58.422270 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:58.422778 (+ 508us) server_negotiation.cc:299] Negotiation successful
0807 02:21:58.422909 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:21:58.423916 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:58.410861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44673 (local address 127.4.221.67:37323)
0807 02:21:58.411356 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0807 02:21:58.411401 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 02:21:58.412672 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:21:58.413262 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:21:58.413269 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:21:58.413628 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:21:58.414194 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:58.414208 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:58.415034 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:58.415037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:58.415913 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:21:58.415924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:21:58.416447 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:21:58.416452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:21:58.416618 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:58.417238 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:21:58.417255 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:21:58.419067 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:21:58.420922 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:58.420930 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:58.420949 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:58.421318 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:58.421616 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:21:58.421622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:21:58.421627 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:21:58.421787 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:21:58.422266 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:21:58.422272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:21:58.422527 (+ 255us) client_negotiation.cc:769] Sending connection context
0807 02:21:58.422723 (+ 196us) client_negotiation.cc:240] Negotiation successful
0807 02:21:58.423065 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260807 02:21:58.424993 7936 heartbeater.cc:347] Connected to a master server at 127.4.221.126:44673
I20260807 02:21:58.425292 7936 heartbeater.cc:464] Registering TS with master...
I20260807 02:21:58.425812 7936 heartbeater.cc:511] Master 127.4.221.126:44673 requested a full tablet report, sending...
I20260807 02:21:58.426901 7464 ts_manager.cc:194] Registered new tserver with Master: 3c14e5e261a44b539a443af50e31867b (127.4.221.67:33861)
I20260807 02:21:58.427505 7464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:37323
I20260807 02:21:58.433080 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 02:21:58.449783 7505 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:21:58.447697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.4.221.126:44673)
0807 02:21:58.447904 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:58.447910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:58.448032 (+ 122us) server_negotiation.cc:407] Connection header received
0807 02:21:58.448538 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:58.448542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:58.448612 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:58.448717 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:58.449605 (+ 888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43642: BlockingRecv error: recv got EOF from 127.0.0.1:43642 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 02:21:58.451002 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 7528
I20260807 02:21:58.458638 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 7666
I20260807 02:21:58.465469 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 7804
I20260807 02:21:58.472594 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 7430
2026-08-07T02:21:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2261 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:21:58 dist-test-slave-69rg krb5kdc[7949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:21:58 dist-test-slave-69rg krb5kdc[7949](info): set up 2 sockets
Aug 07 02:21:58 dist-test-slave-69rg krb5kdc[7949](info): commencing operation
krb5kdc: starting...
W20260807 02:21:59.549592 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:21:59 dist-test-slave-69rg krb5kdc[7949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:21:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:21:59.686046 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:37021
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:37021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:21:59.806879 7962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:21:59.807272 7962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:21:59.807358 7962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:21:59.811661 7962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:21:59.811784 7962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:21:59.811820 7962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:21:59.811861 7962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:21:59.811919 7962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:21:59.817741 7962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:37021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:37021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:21:59.819108 7962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:21:59.820273 7962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 02:21:59.829546 7962 server_base.cc:1034] running on GCE node
W20260807 02:21:59.829499 7967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:59.829488 7971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:21:59.829490 7969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:21:59.830433 7962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:21:59.831634 7962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:21:59.832837 7962 hybrid_clock.cc:648] HybridClock initialized: now 1786069319832810 us; error 45 us; skew 500 ppm
Aug 07 02:21:59 dist-test-slave-69rg krb5kdc[7949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069319, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:21:59.835901 7962 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:21:59.836943 7962 webserver.cc:460] Webserver started at http://127.4.221.126:40699/ using document root <none> and password file <none>
I20260807 02:21:59.837579 7962 fs_manager.cc:359] Metadata directory not provided
I20260807 02:21:59.837683 7962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:21:59.838032 7962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:21:59.840397 7962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "865af0031ebb457a9bdfa4cfc6a9587e"
format_stamp: "Formatted at 2026-08-07 02:21:59 on dist-test-slave-69rg"
server_key: "718034b36ab0eeafc368fbfdbe8b1c89"
server_key_iv: "076ae8c38f578670c84c81c0b954ac65"
server_key_version: "encryptionkey@0"
I20260807 02:21:59.841024 7962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "865af0031ebb457a9bdfa4cfc6a9587e"
format_stamp: "Formatted at 2026-08-07 02:21:59 on dist-test-slave-69rg"
server_key: "718034b36ab0eeafc368fbfdbe8b1c89"
server_key_iv: "076ae8c38f578670c84c81c0b954ac65"
server_key_version: "encryptionkey@0"
I20260807 02:21:59.844972 7962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 02:21:59.847591 7979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:21:59.848838 7962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 02:21:59.848997 7962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "865af0031ebb457a9bdfa4cfc6a9587e"
format_stamp: "Formatted at 2026-08-07 02:21:59 on dist-test-slave-69rg"
server_key: "718034b36ab0eeafc368fbfdbe8b1c89"
server_key_iv: "076ae8c38f578670c84c81c0b954ac65"
server_key_version: "encryptionkey@0"
I20260807 02:21:59.849124 7962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:21:59.863206 7962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:21:59.866634 7962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:21:59.866868 7962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:21:59.874326 7962 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:37021
I20260807 02:21:59.874750 8031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:37021 every 8 connection(s)
I20260807 02:21:59.875378 7962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:21:59.878222 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:21:59.883824 8032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e: Bootstrap starting.
I20260807 02:21:59.884342 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 7962
I20260807 02:21:59.884495 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:21:59.884994 4981 external_mini_cluster.cc:1428] Setting key 5baa1e99409ac485e942d1d794a136a3
I20260807 02:21:59.886991 8032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e: Neither blocks nor log segments found. Creating new log.
I20260807 02:21:59.887825 8032 log.cc:824] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:21:59.889806 8032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e: No bootstrap required, opened a new log
I20260807 02:21:59.893633 8032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865af0031ebb457a9bdfa4cfc6a9587e" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37021 } }
I20260807 02:21:59.893965 8032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:21:59.894027 8032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865af0031ebb457a9bdfa4cfc6a9587e, State: Initialized, Role: FOLLOWER
Aug 07 02:21:59 dist-test-slave-69rg krb5kdc[7949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:21:59.894552 8032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865af0031ebb457a9bdfa4cfc6a9587e" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37021 } }
I20260807 02:21:59.894738 8032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:21:59.894819 8032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:21:59.894932 8032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:21:59.896245 8032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865af0031ebb457a9bdfa4cfc6a9587e" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37021 } }
I20260807 02:21:59.896687 8032 leader_election.cc:302] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 865af0031ebb457a9bdfa4cfc6a9587e; no voters:
I20260807 02:21:59.897059 8032 leader_election.cc:288] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:21:59.897194 8037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:21:59.897435 8037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [term 1 LEADER]: Becoming Leader. State: Replica: 865af0031ebb457a9bdfa4cfc6a9587e, State: Running, Role: LEADER
I20260807 02:21:59.897887 8037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865af0031ebb457a9bdfa4cfc6a9587e" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37021 } }
I20260807 02:21:59.898480 8032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:21:59.899362 8037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 865af0031ebb457a9bdfa4cfc6a9587e. Latest consensus state: current_term: 1 leader_uuid: "865af0031ebb457a9bdfa4cfc6a9587e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865af0031ebb457a9bdfa4cfc6a9587e" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37021 } } }
I20260807 02:21:59.900158 8037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [sys.catalog]: This master's current role is: LEADER
I20260807 02:21:59.900038 8039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "865af0031ebb457a9bdfa4cfc6a9587e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "865af0031ebb457a9bdfa4cfc6a9587e" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37021 } } }
I20260807 02:21:59.900527 8039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e [sys.catalog]: This master's current role is: LEADER
I20260807 02:21:59.901474 8051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:21:59.902293 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:59.886526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.4.221.126:37021)
0807 02:21:59.887005 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:59.887032 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:59.887077 (+ 45us) server_negotiation.cc:407] Connection header received
0807 02:21:59.887839 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:59.887857 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:59.888264 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:59.888616 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:59.889492 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:59.890626 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:59.891506 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:59.892025 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:59.892214 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:59.895259 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:59.895265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:59.895279 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:21:59.895309 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:59.897512 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:59.898145 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:59.898153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:59.898162 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:59.898316 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:21:59.898675 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:21:59.898678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:21:59.898680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:21:59.899227 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:59.900227 (+ 1000us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:59.900435 (+ 208us) server_negotiation.cc:299] Negotiation successful
0807 02:21:59.900787 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":159,"threads_started":1}
I20260807 02:21:59.904868 8051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:21:59.905787 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:21:59.910969 8051 catalog_manager.cc:1347] Generated new cluster ID: f78931599ac14b889f29e8324753c234
I20260807 02:21:59.911057 8051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:21:59.928082 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:21:59.922051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.4.221.126:37021)
0807 02:21:59.922204 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 02:21:59.922210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:21:59.922502 (+ 292us) server_negotiation.cc:407] Connection header received
0807 02:21:59.922638 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:21:59.922642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:21:59.922702 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:21:59.922791 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:21:59.923430 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:59.924251 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:59.925464 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:21:59.925841 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:21:59.925934 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:21:59.926945 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:21:59.926949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:21:59.926952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:21:59.927574 (+ 622us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:21:59.927613 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:21:59.927635 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:21:59.927727 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 02:21:59.927861 (+ 134us) server_negotiation.cc:299] Negotiation successful
0807 02:21:59.927926 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":69}
I20260807 02:21:59.930055 8051 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:21:59.931025 8051 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:21:59.942202 8051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 865af0031ebb457a9bdfa4cfc6a9587e: Generated new TSK 0
I20260807 02:21:59.942992 8051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:21:59.952402 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 7962
2026-08-07T02:21:59Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1477 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:22:00 dist-test-slave-69rg krb5kdc[8065](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:22:00 dist-test-slave-69rg krb5kdc[8065](info): set up 2 sockets
Aug 07 02:22:00 dist-test-slave-69rg krb5kdc[8065](info): commencing operation
krb5kdc: starting...
W20260807 02:22:01.030884 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:22:01 dist-test-slave-69rg krb5kdc[8065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:22:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:22:01.186854 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:42871
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:34943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:42871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:01.316483 8079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:01.316859 8079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:01.316946 8079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:01.321286 8079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:01.321429 8079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:01.321466 8079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:01.321511 8079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:01.321550 8079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:22:01.327381 8079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:34943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:42871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:42871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:01.328680 8079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:01.329913 8079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:01.339200 8088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:01.339172 8085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:01.339520 8079 server_base.cc:1034] running on GCE node
W20260807 02:22:01.339546 8086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:01.340122 8079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:01.341358 8079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:01.342579 8079 hybrid_clock.cc:648] HybridClock initialized: now 1786069321342559 us; error 32 us; skew 500 ppm
Aug 07 02:22:01 dist-test-slave-69rg krb5kdc[8065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069321, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:01.349573 8079 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:22:01.350759 8079 webserver.cc:460] Webserver started at http://127.4.221.126:32883/ using document root <none> and password file <none>
I20260807 02:22:01.351344 8079 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:01.351411 8079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:01.351593 8079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:01.353892 8079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "88ec325b70924955b19caae60420b83c"
format_stamp: "Formatted at 2026-08-07 02:22:01 on dist-test-slave-69rg"
server_key: "7ed93ed713a287385169d2eaafa02158"
server_key_iv: "4d30ca2fe8cf3220b575e78e0cf37ff2"
server_key_version: "encryptionkey@0"
I20260807 02:22:01.354501 8079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "88ec325b70924955b19caae60420b83c"
format_stamp: "Formatted at 2026-08-07 02:22:01 on dist-test-slave-69rg"
server_key: "7ed93ed713a287385169d2eaafa02158"
server_key_iv: "4d30ca2fe8cf3220b575e78e0cf37ff2"
server_key_version: "encryptionkey@0"
I20260807 02:22:01.358954 8079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:22:01.361507 8095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:01.362939 8079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:22:01.363068 8079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "88ec325b70924955b19caae60420b83c"
format_stamp: "Formatted at 2026-08-07 02:22:01 on dist-test-slave-69rg"
server_key: "7ed93ed713a287385169d2eaafa02158"
server_key_iv: "4d30ca2fe8cf3220b575e78e0cf37ff2"
server_key_version: "encryptionkey@0"
I20260807 02:22:01.363173 8079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:01.390436 8079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:01.394002 8079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:01.394227 8079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:01.402281 8079 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:42871
I20260807 02:22:01.402654 8148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:42871 every 8 connection(s)
I20260807 02:22:01.403407 8079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:01.406626 8149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:01.407186 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 8079
I20260807 02:22:01.407336 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:22:01.407689 4981 external_mini_cluster.cc:1428] Setting key 54f314fd3988ad127b43f8c0858a0b72
I20260807 02:22:01.413407 8149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c: Bootstrap starting.
I20260807 02:22:01.416297 8149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:01.417224 8149 log.cc:824] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c: Log is configured to *not* fsync() on all Append() calls
Aug 07 02:22:01 dist-test-slave-69rg krb5kdc[8065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:01.419920 8149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c: No bootstrap required, opened a new log
I20260807 02:22:01.423789 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:01.409274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.4.221.126:42871)
0807 02:22:01.409741 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:01.409757 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:01.409794 (+ 37us) server_negotiation.cc:407] Connection header received
0807 02:22:01.410535 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:01.410553 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:01.410896 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:01.411192 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:01.412252 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:01.413250 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:01.414434 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:01.414938 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:01.415080 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:01.418209 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:01.418216 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:01.418234 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:01.418265 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:01.420324 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:01.420777 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:01.420783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:01.420789 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:01.420932 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:01.421195 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:01.421199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:01.421200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:01.421714 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:01.421949 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:01.422329 (+ 380us) server_negotiation.cc:299] Negotiation successful
0807 02:22:01.422722 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":160,"threads_started":1}
I20260807 02:22:01.424932 8149 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ec325b70924955b19caae60420b83c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 42871 } }
I20260807 02:22:01.425235 8149 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:01.425304 8149 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88ec325b70924955b19caae60420b83c, State: Initialized, Role: FOLLOWER
I20260807 02:22:01.425729 8149 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ec325b70924955b19caae60420b83c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 42871 } }
I20260807 02:22:01.425909 8149 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:01.425977 8149 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:01.426077 8149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:01.427167 8149 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ec325b70924955b19caae60420b83c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 42871 } }
I20260807 02:22:01.427487 8149 leader_election.cc:302] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88ec325b70924955b19caae60420b83c; no voters:
I20260807 02:22:01.427824 8149 leader_election.cc:288] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:22:01.428049 8154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:01.428411 8154 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [term 1 LEADER]: Becoming Leader. State: Replica: 88ec325b70924955b19caae60420b83c, State: Running, Role: LEADER
I20260807 02:22:01.428874 8154 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ec325b70924955b19caae60420b83c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 42871 } }
I20260807 02:22:01.429415 8149 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:01.430783 8156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88ec325b70924955b19caae60420b83c. Latest consensus state: current_term: 1 leader_uuid: "88ec325b70924955b19caae60420b83c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ec325b70924955b19caae60420b83c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 42871 } } }
I20260807 02:22:01.430927 8156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:01.431254 8155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88ec325b70924955b19caae60420b83c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88ec325b70924955b19caae60420b83c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 42871 } } }
I20260807 02:22:01.431362 8155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c [sys.catalog]: This master's current role is: LEADER
W20260807 02:22:01.434741 8169 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:22:01.434844 8169 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:22:01.434912 8163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:01.435575 8163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:01.441987 8163 catalog_manager.cc:1347] Generated new cluster ID: cab68698b5bf4c948d70c9c22f29e90f
I20260807 02:22:01.442078 8163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:01.455427 8163 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:22:01.456689 8163 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:01.462311 8163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88ec325b70924955b19caae60420b83c: Generated new TSK 0
I20260807 02:22:01.463027 8163 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:22:01.502869 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:42871
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:34943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:01.623099 8175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:01.623420 8175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:01.623466 8175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:01.627653 8175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:01.627748 8175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:01.627813 8175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:22:01.633809 8175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:34943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:42871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:01.635044 8175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:01.636070 8175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:01.646375 8182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:01.646476 8184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:01.646598 8175 server_base.cc:1034] running on GCE node
W20260807 02:22:01.646375 8181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:01.647310 8175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:01.647944 8175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:01.649112 8175 hybrid_clock.cc:648] HybridClock initialized: now 1786069321649103 us; error 47 us; skew 500 ppm
Aug 07 02:22:01 dist-test-slave-69rg krb5kdc[8065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069321, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:01.651947 8175 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:22:01.652810 8175 webserver.cc:460] Webserver started at http://127.4.221.65:33845/ using document root <none> and password file <none>
I20260807 02:22:01.653350 8175 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:01.653417 8175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:01.653589 8175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:01.655717 8175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "57f2d5bb825c426c8cfbae2b52fe9c2a"
format_stamp: "Formatted at 2026-08-07 02:22:01 on dist-test-slave-69rg"
server_key: "08dab0c6f87f6abc861ffe82e49873fd"
server_key_iv: "f1f323ed67e5a5bb4728ea443e3c3c5b"
server_key_version: "encryptionkey@0"
I20260807 02:22:01.656242 8175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "57f2d5bb825c426c8cfbae2b52fe9c2a"
format_stamp: "Formatted at 2026-08-07 02:22:01 on dist-test-slave-69rg"
server_key: "08dab0c6f87f6abc861ffe82e49873fd"
server_key_iv: "f1f323ed67e5a5bb4728ea443e3c3c5b"
server_key_version: "encryptionkey@0"
I20260807 02:22:01.660285 8175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:22:01.663317 8191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:01.664682 8175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:22:01.664809 8175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "57f2d5bb825c426c8cfbae2b52fe9c2a"
format_stamp: "Formatted at 2026-08-07 02:22:01 on dist-test-slave-69rg"
server_key: "08dab0c6f87f6abc861ffe82e49873fd"
server_key_iv: "f1f323ed67e5a5bb4728ea443e3c3c5b"
server_key_version: "encryptionkey@0"
I20260807 02:22:01.664916 8175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:01.675522 8175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:01.679111 8175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:01.679348 8175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:01.679984 8175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:01.680869 8175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:01.680919 8175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:01.680989 8175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:01.681020 8175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:01.690405 8175 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:40495
I20260807 02:22:01.690442 8307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:40495 every 8 connection(s)
I20260807 02:22:01.691442 8175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:22:01.700743 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 8175
I20260807 02:22:01.700922 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:22:01.701265 4981 external_mini_cluster.cc:1428] Setting key 22f09aecd2554096ac35d4a8ceb259d7
Aug 07 02:22:01 dist-test-slave-69rg krb5kdc[8065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069321, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:01.707115 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:01.693567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:56647 (local address 127.4.221.126:42871)
0807 02:22:01.693720 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:01.693724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:01.694686 (+ 962us) server_negotiation.cc:407] Connection header received
0807 02:22:01.696177 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:01.696181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:01.696244 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:01.696365 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:01.697884 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:01.698465 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:01.699711 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:01.699956 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:01.700079 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:01.703212 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:01.703215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:01.703217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:01.703244 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:01.704801 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:01.705363 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:01.705369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:01.705370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:01.705461 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:01.705858 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:01.705865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:01.705867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:01.706066 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:01.706190 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:01.706866 (+ 676us) server_negotiation.cc:299] Negotiation successful
0807 02:22:01.706986 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 02:22:01.708461 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:01.694041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:42871 (local address 127.4.221.65:56647)
0807 02:22:01.694521 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:01.694567 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:01.695945 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:01.696568 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:01.696575 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:01.697110 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:01.697668 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:01.697683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:01.698611 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:01.698615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:01.699564 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:01.699574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:01.700128 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:01.700132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:01.700324 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:01.701090 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:01.701107 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:01.703023 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:01.704931 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:01.704938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:01.704955 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:01.705252 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:01.705570 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:01.705573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:01.705575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:01.705740 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:01.706181 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:01.706188 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:01.706556 (+ 368us) client_negotiation.cc:769] Sending connection context
0807 02:22:01.706797 (+ 241us) client_negotiation.cc:240] Negotiation successful
0807 02:22:01.707197 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":134,"threads_started":1}
I20260807 02:22:01.709873 8308 heartbeater.cc:347] Connected to a master server at 127.4.221.126:42871
I20260807 02:22:01.710170 8308 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:01.710844 8308 heartbeater.cc:511] Master 127.4.221.126:42871 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
I20260807 02:22:01.712527 8112 ts_manager.cc:194] Registered new tserver with Master: 57f2d5bb825c426c8cfbae2b52fe9c2a (127.4.221.65:40495)
I20260807 02:22:01.713925 8112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:56647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:22:01.735219 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:42871
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:34943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:01.851151 8313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:01.851459 8313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:01.851509 8313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:01.855893 8313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:01.855978 8313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:01.856077 8313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:22:01.862249 8313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:34943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:42871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:01.863453 8313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:01.864537 8313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:01.874889 8319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:01.874894 8318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:01.875224 8313 server_base.cc:1034] running on GCE node
W20260807 02:22:01.874894 8321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:01.875744 8313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:01.876466 8313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:01.877661 8313 hybrid_clock.cc:648] HybridClock initialized: now 1786069321877644 us; error 28 us; skew 500 ppm
Aug 07 02:22:01 dist-test-slave-69rg krb5kdc[8065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069321, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:01.880656 8313 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:22:01.881631 8313 webserver.cc:460] Webserver started at http://127.4.221.66:37755/ using document root <none> and password file <none>
I20260807 02:22:01.882288 8313 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:01.882382 8313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:01.882627 8313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:01.884847 8313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "6886fb074ce94141b4b37872889c0c77"
format_stamp: "Formatted at 2026-08-07 02:22:01 on dist-test-slave-69rg"
server_key: "ba9620a289f602dcab150bf1d5ea20ac"
server_key_iv: "7bc93e57d2c61d910625b53c985d7f88"
server_key_version: "encryptionkey@0"
I20260807 02:22:01.885489 8313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "6886fb074ce94141b4b37872889c0c77"
format_stamp: "Formatted at 2026-08-07 02:22:01 on dist-test-slave-69rg"
server_key: "ba9620a289f602dcab150bf1d5ea20ac"
server_key_iv: "7bc93e57d2c61d910625b53c985d7f88"
server_key_version: "encryptionkey@0"
I20260807 02:22:01.889618 8313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260807 02:22:01.892289 8328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:01.893656 8313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:22:01.893810 8313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "6886fb074ce94141b4b37872889c0c77"
format_stamp: "Formatted at 2026-08-07 02:22:01 on dist-test-slave-69rg"
server_key: "ba9620a289f602dcab150bf1d5ea20ac"
server_key_iv: "7bc93e57d2c61d910625b53c985d7f88"
server_key_version: "encryptionkey@0"
I20260807 02:22:01.894009 8313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:01.941490 8313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:01.945151 8313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:01.945384 8313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:01.946110 8313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:01.947008 8313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:01.947079 8313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:01.947149 8313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:01.947196 8313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:01.956578 8313 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:44979
I20260807 02:22:01.956633 8443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:44979 every 8 connection(s)
I20260807 02:22:01.957635 8313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:22:01.964133 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 8313
I20260807 02:22:01.964291 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:22:01.964592 4981 external_mini_cluster.cc:1428] Setting key 90bc0a88a3dc28f6813f21dbffc00a86
Aug 07 02:22:01 dist-test-slave-69rg krb5kdc[8065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069321, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:01.973589 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:01.959607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:34565 (local address 127.4.221.126:42871)
0807 02:22:01.959772 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:01.959777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:01.960794 (+ 1017us) server_negotiation.cc:407] Connection header received
0807 02:22:01.962541 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:01.962544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:01.962610 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:01.962718 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:01.963997 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:01.964704 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:01.966004 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:01.966250 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:01.966374 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:01.969275 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:01.969279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:01.969282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:01.969323 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:01.970868 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:01.971615 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:01.971618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:01.971619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:01.971716 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:01.972210 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:01.972213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:01.972214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:01.972428 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:01.972520 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:01.973317 (+ 797us) server_negotiation.cc:299] Negotiation successful
0807 02:22:01.973454 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 02:22:01.975008 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:01.960049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:42871 (local address 127.4.221.66:34565)
0807 02:22:01.960615 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:01.960660 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:01.962289 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:01.962915 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:01.962923 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:01.963328 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:01.963835 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:01.963850 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:01.964843 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:01.964847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:01.965804 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:01.965874 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:01.966433 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:01.966438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:01.966640 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:01.967412 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:01.967430 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:01.969081 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:01.971015 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:01.971026 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:01.971051 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:01.971495 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:01.971844 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:01.971850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:01.971854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:01.972076 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:01.972555 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:01.972565 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:01.972997 (+ 432us) client_negotiation.cc:769] Sending connection context
0807 02:22:01.973260 (+ 263us) client_negotiation.cc:240] Negotiation successful
0807 02:22:01.973684 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":157,"threads_started":1}
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
I20260807 02:22:01.976559 8444 heartbeater.cc:347] Connected to a master server at 127.4.221.126:42871
I20260807 02:22:01.976858 8444 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:01.977447 8444 heartbeater.cc:511] Master 127.4.221.126:42871 requested a full tablet report, sending...
I20260807 02:22:01.978817 8112 ts_manager.cc:194] Registered new tserver with Master: 6886fb074ce94141b4b37872889c0c77 (127.4.221.66:44979)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
I20260807 02:22:01.979591 8112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:34565
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:22:01.997506 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:42871
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:34943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:02.119961 8449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:02.120378 8449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:02.120465 8449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:02.124835 8449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:02.124974 8449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:02.125079 8449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:22:02.131096 8449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:34943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:42871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:02.132472 8449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:02.133603 8449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:02.144109 8455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:02.144120 8457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:02.144109 8454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:02.144434 8449 server_base.cc:1034] running on GCE node
I20260807 02:22:02.145004 8449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:02.145757 8449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:02.146975 8449 hybrid_clock.cc:648] HybridClock initialized: now 1786069322146960 us; error 26 us; skew 500 ppm
Aug 07 02:22:02 dist-test-slave-69rg krb5kdc[8065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:02.149997 8449 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:22:02.151031 8449 webserver.cc:460] Webserver started at http://127.4.221.67:34251/ using document root <none> and password file <none>
I20260807 02:22:02.151724 8449 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:02.151795 8449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:02.151975 8449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:02.154249 8449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "d323a1f45a0f41618735da8ab17513b4"
format_stamp: "Formatted at 2026-08-07 02:22:02 on dist-test-slave-69rg"
server_key: "38e9b1a2774c66056ae23b1af079d812"
server_key_iv: "bf03ee604b56b8a5af9d12923831dcea"
server_key_version: "encryptionkey@0"
I20260807 02:22:02.154821 8449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "d323a1f45a0f41618735da8ab17513b4"
format_stamp: "Formatted at 2026-08-07 02:22:02 on dist-test-slave-69rg"
server_key: "38e9b1a2774c66056ae23b1af079d812"
server_key_iv: "bf03ee604b56b8a5af9d12923831dcea"
server_key_version: "encryptionkey@0"
I20260807 02:22:02.158923 8449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:22:02.161442 8464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:02.162859 8449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:22:02.163012 8449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "d323a1f45a0f41618735da8ab17513b4"
format_stamp: "Formatted at 2026-08-07 02:22:02 on dist-test-slave-69rg"
server_key: "38e9b1a2774c66056ae23b1af079d812"
server_key_iv: "bf03ee604b56b8a5af9d12923831dcea"
server_key_version: "encryptionkey@0"
I20260807 02:22:02.163128 8449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:02.210754 8449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:02.214269 8449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:02.214485 8449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:02.215116 8449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:02.216013 8449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:02.216075 8449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:02.216118 8449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:02.216135 8449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:02.225294 8449 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:34521
I20260807 02:22:02.225319 8580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:34521 every 8 connection(s)
I20260807 02:22:02.226349 8449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
Aug 07 02:22:02 dist-test-slave-69rg krb5kdc[8065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069322, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:02.236548 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 8449
I20260807 02:22:02.236691 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:22:02.237000 4981 external_mini_cluster.cc:1428] Setting key 12c39b885d664c2f40c81130da53f238
I20260807 02:22:02.241248 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:02.228255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:49543 (local address 127.4.221.126:42871)
0807 02:22:02.228412 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:02.228417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:02.229460 (+ 1043us) server_negotiation.cc:407] Connection header received
0807 02:22:02.230996 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:02.230999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:02.231061 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:02.231143 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:02.232412 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:02.232992 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:02.234242 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:02.234473 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:02.234556 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:02.237245 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:02.237249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:02.237251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:02.237280 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:02.238977 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:02.239633 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:02.239638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:02.239640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:02.239730 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:02.240094 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:02.240099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:02.240101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:02.240314 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:02.240422 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:02.240948 (+ 526us) server_negotiation.cc:299] Negotiation successful
0807 02:22:02.241080 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:22:02.242281 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:02.228689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:42871 (local address 127.4.221.67:49543)
0807 02:22:02.229276 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:02.229330 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:02.230740 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:02.231343 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:02.231350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:02.231732 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:02.232231 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:02.232246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:02.233168 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:02.233170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:02.234124 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:02.234134 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:02.234613 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:02.234616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:02.234763 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:02.235362 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:02.235380 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:02.237025 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:02.239109 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:02.239116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:02.239132 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:02.239528 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:02.239836 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:02.239839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:02.239841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:02.239991 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:02.240433 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:02.240442 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:02.240709 (+ 267us) client_negotiation.cc:769] Sending connection context
0807 02:22:02.240907 (+ 198us) client_negotiation.cc:240] Negotiation successful
0807 02:22:02.241252 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":121,"threads_started":1}
I20260807 02:22:02.243904 8581 heartbeater.cc:347] Connected to a master server at 127.4.221.126:42871
I20260807 02:22:02.244246 8581 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:02.244863 8581 heartbeater.cc:511] Master 127.4.221.126:42871 requested a full tablet report, sending...
I20260807 02:22:02.246331 8112 ts_manager.cc:194] Registered new tserver with Master: d323a1f45a0f41618735da8ab17513b4 (127.4.221.67:34521)
I20260807 02:22:02.247129 8112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:49543
I20260807 02:22:02.251842 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 02:22:02 dist-test-slave-69rg krb5kdc[8065](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069322, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 07 02:22:02 dist-test-slave-69rg krb5kdc[8065](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069322, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:02.281173 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:02.271967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41280 (local address 127.4.221.126:42871)
0807 02:22:02.272161 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:02.272167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:02.272353 (+ 186us) server_negotiation.cc:407] Connection header received
0807 02:22:02.272552 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:02.272561 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:02.272608 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:02.272700 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:02.273497 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:02.274150 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:02.275032 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:02.275262 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:02.275377 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:02.278084 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:02.278087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:02.278090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:02.278134 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:02.279587 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:02.280010 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:02.280012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:02.280014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:02.280090 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:02.280359 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:02.280362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:02.280363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:02.280604 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:02.280722 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:02.280934 (+ 212us) server_negotiation.cc:299] Negotiation successful
0807 02:22:02.281041 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260807 02:22:02.281733 8112 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41280
I20260807 02:22:02.282832 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 8175
I20260807 02:22:02.290820 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 8313
I20260807 02:22:02.299530 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 8449
I20260807 02:22:02.306527 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 8079
2026-08-07T02:22:02Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2356 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:22:02 dist-test-slave-69rg krb5kdc[8593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:22:02 dist-test-slave-69rg krb5kdc[8593](info): set up 2 sockets
Aug 07 02:22:02 dist-test-slave-69rg krb5kdc[8593](info): commencing operation
krb5kdc: starting...
W20260807 02:22:03.369941 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:22:03 dist-test-slave-69rg krb5kdc[8593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:22:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:22:03.504709 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:35641
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:39695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:35641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:03.634260 8606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:03.634701 8606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:03.634783 8606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:03.639189 8606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:03.639319 8606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:03.639357 8606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:03.639402 8606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:03.639436 8606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:22:03.645684 8606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:39695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:35641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:35641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:03.647356 8606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:03.648788 8606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:03.659327 8614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:03.659514 8606 server_base.cc:1034] running on GCE node
W20260807 02:22:03.659293 8612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:03.659368 8611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:03.660315 8606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:03.661931 8606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:03.663187 8606 hybrid_clock.cc:648] HybridClock initialized: now 1786069323663166 us; error 39 us; skew 500 ppm
Aug 07 02:22:03 dist-test-slave-69rg krb5kdc[8593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069323, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:03.666883 8606 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:22:03.668105 8606 webserver.cc:460] Webserver started at http://127.4.221.126:41405/ using document root <none> and password file <none>
I20260807 02:22:03.668854 8606 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:03.668959 8606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:03.669250 8606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:03.671962 8606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "a800fc846d9e4dc4ba950b07a2319520"
format_stamp: "Formatted at 2026-08-07 02:22:03 on dist-test-slave-69rg"
server_key: "2f137241b8671874df8b182db3a76d29"
server_key_iv: "2a3520afec85002331918afb943a59e4"
server_key_version: "encryptionkey@0"
I20260807 02:22:03.672716 8606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "a800fc846d9e4dc4ba950b07a2319520"
format_stamp: "Formatted at 2026-08-07 02:22:03 on dist-test-slave-69rg"
server_key: "2f137241b8671874df8b182db3a76d29"
server_key_iv: "2a3520afec85002331918afb943a59e4"
server_key_version: "encryptionkey@0"
I20260807 02:22:03.677549 8606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260807 02:22:03.680614 8621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:03.682298 8606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:22:03.682492 8606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "a800fc846d9e4dc4ba950b07a2319520"
format_stamp: "Formatted at 2026-08-07 02:22:03 on dist-test-slave-69rg"
server_key: "2f137241b8671874df8b182db3a76d29"
server_key_iv: "2a3520afec85002331918afb943a59e4"
server_key_version: "encryptionkey@0"
I20260807 02:22:03.682649 8606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:03.696331 8606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:03.700172 8606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:03.700484 8606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:03.709489 8606 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:35641
I20260807 02:22:03.709509 8675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:35641 every 8 connection(s)
I20260807 02:22:03.710804 8606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:03.713312 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 8606
I20260807 02:22:03.713459 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:22:03.713855 4981 external_mini_cluster.cc:1428] Setting key 0539586b924d325ef5a13207998d4703
I20260807 02:22:03.714818 8676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:03.723212 8676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520: Bootstrap starting.
Aug 07 02:22:03 dist-test-slave-69rg krb5kdc[8593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:03.727337 8676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:03.728539 8676 log.cc:824] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:03.731329 8676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520: No bootstrap required, opened a new log
I20260807 02:22:03.731961 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:03.715671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.4.221.126:35641)
0807 02:22:03.716203 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:03.716223 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:03.716266 (+ 43us) server_negotiation.cc:407] Connection header received
0807 02:22:03.717049 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:03.717071 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:03.717449 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:03.717772 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:03.718932 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:03.720067 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:03.721326 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:03.721919 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:03.722054 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:03.725874 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:03.725883 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:03.725905 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:03.725950 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:03.728412 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:03.728958 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:03.728971 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:03.728977 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:03.729112 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:03.729434 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:03.729439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:03.729441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:03.730028 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:03.730272 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:03.730602 (+ 330us) server_negotiation.cc:299] Negotiation successful
0807 02:22:03.730973 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":154,"threads_started":1}
I20260807 02:22:03.736323 8676 raft_consensus.cc:348] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a800fc846d9e4dc4ba950b07a2319520" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35641 } }
I20260807 02:22:03.736697 8676 raft_consensus.cc:374] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:03.736768 8676 raft_consensus.cc:729] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a800fc846d9e4dc4ba950b07a2319520, State: Initialized, Role: FOLLOWER
I20260807 02:22:03.737339 8676 consensus_queue.cc:260] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a800fc846d9e4dc4ba950b07a2319520" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35641 } }
I20260807 02:22:03.737526 8676 raft_consensus.cc:388] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:03.737623 8676 raft_consensus.cc:482] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:03.737764 8676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:03.739135 8676 raft_consensus.cc:504] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a800fc846d9e4dc4ba950b07a2319520" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35641 } }
I20260807 02:22:03.739581 8676 leader_election.cc:302] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a800fc846d9e4dc4ba950b07a2319520; no voters:
I20260807 02:22:03.740083 8676 leader_election.cc:288] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:22:03.740185 8681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:03.740497 8681 raft_consensus.cc:686] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [term 1 LEADER]: Becoming Leader. State: Replica: a800fc846d9e4dc4ba950b07a2319520, State: Running, Role: LEADER
I20260807 02:22:03.740867 8681 consensus_queue.cc:237] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a800fc846d9e4dc4ba950b07a2319520" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35641 } }
I20260807 02:22:03.741981 8676 sys_catalog.cc:564] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:03.742494 8683 sys_catalog.cc:455] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a800fc846d9e4dc4ba950b07a2319520. Latest consensus state: current_term: 1 leader_uuid: "a800fc846d9e4dc4ba950b07a2319520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a800fc846d9e4dc4ba950b07a2319520" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35641 } } }
I20260807 02:22:03.742635 8683 sys_catalog.cc:458] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:03.742980 8682 sys_catalog.cc:455] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a800fc846d9e4dc4ba950b07a2319520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a800fc846d9e4dc4ba950b07a2319520" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35641 } } }
I20260807 02:22:03.743100 8682 sys_catalog.cc:458] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:03.743296 8685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:03.747256 8685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:03.755530 8685 catalog_manager.cc:1347] Generated new cluster ID: b9b9f20e6c7f430dbef56f1f754335dc
I20260807 02:22:03.755683 8685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:03.771643 8685 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:22:03.773895 8685 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:03.784454 8685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a800fc846d9e4dc4ba950b07a2319520: Generated new TSK 0
I20260807 02:22:03.785542 8685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:22:03.831852 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:35641
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:39695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:03.987533 8703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:03.987972 8703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:03.988075 8703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:03.992427 8703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:03.992619 8703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:03.992790 8703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:22:03.998980 8703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:39695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:35641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:04.000520 8703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:04.001922 8703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:04.014832 8709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:04.014838 8711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:04.014910 8708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:04.014885 8703 server_base.cc:1034] running on GCE node
I20260807 02:22:04.015781 8703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:04.016651 8703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:04.017982 8703 hybrid_clock.cc:648] HybridClock initialized: now 1786069324017957 us; error 40 us; skew 500 ppm
Aug 07 02:22:04 dist-test-slave-69rg krb5kdc[8593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069324, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:04.021500 8703 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:22:04.022696 8703 webserver.cc:460] Webserver started at http://127.4.221.65:39033/ using document root <none> and password file <none>
I20260807 02:22:04.023370 8703 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:04.023438 8703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:04.023648 8703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:04.026365 8703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "50d9f37ed12d44c198fc2fa7be428fdb"
format_stamp: "Formatted at 2026-08-07 02:22:04 on dist-test-slave-69rg"
server_key: "e21cbcb6cd23bcba1d84a7dbf9051e34"
server_key_iv: "d867ae07391f5b21ec194644f58d1a53"
server_key_version: "encryptionkey@0"
I20260807 02:22:04.027071 8703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "50d9f37ed12d44c198fc2fa7be428fdb"
format_stamp: "Formatted at 2026-08-07 02:22:04 on dist-test-slave-69rg"
server_key: "e21cbcb6cd23bcba1d84a7dbf9051e34"
server_key_iv: "d867ae07391f5b21ec194644f58d1a53"
server_key_version: "encryptionkey@0"
I20260807 02:22:04.031857 8703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:22:04.035189 8718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:04.036985 8703 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:22:04.037196 8703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "50d9f37ed12d44c198fc2fa7be428fdb"
format_stamp: "Formatted at 2026-08-07 02:22:04 on dist-test-slave-69rg"
server_key: "e21cbcb6cd23bcba1d84a7dbf9051e34"
server_key_iv: "d867ae07391f5b21ec194644f58d1a53"
server_key_version: "encryptionkey@0"
I20260807 02:22:04.037358 8703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:04.056806 8703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:04.062292 8703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:04.062608 8703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:04.063757 8703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:04.065251 8703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:04.065318 8703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:04.065402 8703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:04.065438 8703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:04.076952 8703 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:40819
I20260807 02:22:04.076989 8835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:40819 every 8 connection(s)
I20260807 02:22:04.078235 8703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:22:04.083516 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 8703
I20260807 02:22:04.083659 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:22:04.084172 4981 external_mini_cluster.cc:1428] Setting key c836969ce709969037ae8df1d32f341e
Aug 07 02:22:04 dist-test-slave-69rg krb5kdc[8593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069324, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:04.096731 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:04.080730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:49847 (local address 127.4.221.126:35641)
0807 02:22:04.080948 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:04.080954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:04.082235 (+ 1281us) server_negotiation.cc:407] Connection header received
0807 02:22:04.083985 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:04.083991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:04.084076 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:04.084205 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:04.086134 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.087190 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.088659 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.089153 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.089266 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:04.092536 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:04.092540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:04.092542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:04.092570 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:04.094303 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:04.094901 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:04.094904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:04.094906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:04.094992 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:04.095372 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:04.095376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:04.095378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:04.095606 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:04.095720 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:04.096352 (+ 632us) server_negotiation.cc:299] Negotiation successful
0807 02:22:04.096522 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
I20260807 02:22:04.097952 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:04.081297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:35641 (local address 127.4.221.65:49847)
0807 02:22:04.082036 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:04.082097 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:04.083666 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:04.084441 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:04.084450 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:04.085054 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:04.085921 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:04.085958 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.087331 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.087334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:04.088528 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:04.088539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.089357 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.089362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:04.089671 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:04.090526 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:04.090554 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:04.092333 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:04.094433 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:04.094440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:04.094461 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:04.094775 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:04.095097 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:04.095101 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:04.095103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:04.095265 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:04.095716 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:04.095724 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:04.096043 (+ 319us) client_negotiation.cc:769] Sending connection context
0807 02:22:04.096283 (+ 240us) client_negotiation.cc:240] Negotiation successful
0807 02:22:04.096716 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":171,"threads_started":1}
I20260807 02:22:04.099740 8836 heartbeater.cc:347] Connected to a master server at 127.4.221.126:35641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
I20260807 02:22:04.100126 8836 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:04.101095 8836 heartbeater.cc:511] Master 127.4.221.126:35641 requested a full tablet report, sending...
I20260807 02:22:04.103621 8638 ts_manager.cc:194] Registered new tserver with Master: 50d9f37ed12d44c198fc2fa7be428fdb (127.4.221.65:40819)
I20260807 02:22:04.105243 8638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:49847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:22:04.123556 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:35641
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:39695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:04.260241 8841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:04.260574 8841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:04.260624 8841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:04.265177 8841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:04.265254 8841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:04.265352 8841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:22:04.271265 8841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:39695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:35641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:04.272763 8841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:04.274106 8841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:04.285786 8847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:04.285794 8846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:04.285862 8849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:04.286180 8841 server_base.cc:1034] running on GCE node
I20260807 02:22:04.286753 8841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:04.287587 8841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:04.288754 8841 hybrid_clock.cc:648] HybridClock initialized: now 1786069324288709 us; error 83 us; skew 500 ppm
Aug 07 02:22:04 dist-test-slave-69rg krb5kdc[8593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069324, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:04.292200 8841 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:22:04.293351 8841 webserver.cc:460] Webserver started at http://127.4.221.66:39755/ using document root <none> and password file <none>
I20260807 02:22:04.294142 8841 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:04.294245 8841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:04.294595 8841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:04.297444 8841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "e4c7946a3b154b2289f03f2329f5987f"
format_stamp: "Formatted at 2026-08-07 02:22:04 on dist-test-slave-69rg"
server_key: "5e5a671091eb25ef1f11d728e8e2bdd7"
server_key_iv: "7cdc2215abce32941ef86eb2bac4c923"
server_key_version: "encryptionkey@0"
I20260807 02:22:04.298188 8841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "e4c7946a3b154b2289f03f2329f5987f"
format_stamp: "Formatted at 2026-08-07 02:22:04 on dist-test-slave-69rg"
server_key: "5e5a671091eb25ef1f11d728e8e2bdd7"
server_key_iv: "7cdc2215abce32941ef86eb2bac4c923"
server_key_version: "encryptionkey@0"
I20260807 02:22:04.302928 8841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 02:22:04.306114 8856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:04.307631 8841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 02:22:04.307785 8841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "e4c7946a3b154b2289f03f2329f5987f"
format_stamp: "Formatted at 2026-08-07 02:22:04 on dist-test-slave-69rg"
server_key: "5e5a671091eb25ef1f11d728e8e2bdd7"
server_key_iv: "7cdc2215abce32941ef86eb2bac4c923"
server_key_version: "encryptionkey@0"
I20260807 02:22:04.307921 8841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:04.323249 8841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:04.327023 8841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:04.327306 8841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:04.328086 8841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:04.329156 8841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:04.329234 8841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:04.329313 8841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:04.329357 8841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:04.340683 8841 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:36251
I20260807 02:22:04.340713 8974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:36251 every 8 connection(s)
I20260807 02:22:04.341986 8841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:22:04.344012 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 8841
I20260807 02:22:04.344159 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:22:04.344570 4981 external_mini_cluster.cc:1428] Setting key 74704d3abbc10fc5353bfd02c2c897fd
Aug 07 02:22:04 dist-test-slave-69rg krb5kdc[8593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069324, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
I20260807 02:22:04.360447 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:04.344752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:58547 (local address 127.4.221.126:35641)
0807 02:22:04.344949 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:04.344955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:04.346194 (+ 1239us) server_negotiation.cc:407] Connection header received
0807 02:22:04.347857 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:04.347863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:04.347940 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:04.348048 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:04.349484 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.350461 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.351767 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.352033 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.352185 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:04.355601 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:04.355607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:04.355614 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:04.355661 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:04.357586 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:04.358352 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:04.358357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:04.358358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:04.358454 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:04.358878 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:04.358881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:04.358883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:04.359309 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:04.359401 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:04.360088 (+ 687us) server_negotiation.cc:299] Negotiation successful
0807 02:22:04.360234 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
I20260807 02:22:04.361857 8977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:04.345377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:35641 (local address 127.4.221.66:58547)
0807 02:22:04.346008 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:04.346072 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:04.347596 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:04.348242 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:04.348250 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:04.348708 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:04.349310 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:04.349325 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.350613 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.350616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:04.351624 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:04.351634 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.352243 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.352248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:04.352462 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:04.353229 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:04.353250 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:04.355355 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:04.357744 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:04.357751 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:04.357768 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:04.358235 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:04.358562 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:04.358565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:04.358567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:04.358751 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:04.359448 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:04.359455 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:04.359789 (+ 334us) client_negotiation.cc:769] Sending connection context
0807 02:22:04.360007 (+ 218us) client_negotiation.cc:240] Negotiation successful
0807 02:22:04.360375 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":180,"threads_started":1}
I20260807 02:22:04.363623 8975 heartbeater.cc:347] Connected to a master server at 127.4.221.126:35641
I20260807 02:22:04.363968 8975 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:04.364734 8975 heartbeater.cc:511] Master 127.4.221.126:35641 requested a full tablet report, sending...
I20260807 02:22:04.366286 8638 ts_manager.cc:194] Registered new tserver with Master: e4c7946a3b154b2289f03f2329f5987f (127.4.221.66:36251)
I20260807 02:22:04.367079 8638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:58547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:22:04.382217 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:35641
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:39695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:04.516654 8980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:04.516983 8980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:04.517032 8980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:04.521494 8980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:04.521574 8980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:04.521637 8980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:22:04.527205 8980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:39695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:35641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:04.528561 8980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:04.529865 8980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:04.542188 8985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:04.542188 8988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:04.542362 8986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:04.542719 8980 server_base.cc:1034] running on GCE node
I20260807 02:22:04.543268 8980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:04.544101 8980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:04.545275 8980 hybrid_clock.cc:648] HybridClock initialized: now 1786069324545230 us; error 67 us; skew 500 ppm
Aug 07 02:22:04 dist-test-slave-69rg krb5kdc[8593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069324, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:04.548878 8980 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:22:04.550114 8980 webserver.cc:460] Webserver started at http://127.4.221.67:36213/ using document root <none> and password file <none>
I20260807 02:22:04.550804 8980 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:04.550869 8980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:04.551121 8980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:04.553710 8980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "8b8faebf9afc4e039d62023404f46902"
format_stamp: "Formatted at 2026-08-07 02:22:04 on dist-test-slave-69rg"
server_key: "02408fdf5266321672afb7cf2d45779f"
server_key_iv: "d81ed5ee9cdcb16e96bf14cd3db23b2a"
server_key_version: "encryptionkey@0"
I20260807 02:22:04.554423 8980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "8b8faebf9afc4e039d62023404f46902"
format_stamp: "Formatted at 2026-08-07 02:22:04 on dist-test-slave-69rg"
server_key: "02408fdf5266321672afb7cf2d45779f"
server_key_iv: "d81ed5ee9cdcb16e96bf14cd3db23b2a"
server_key_version: "encryptionkey@0"
I20260807 02:22:04.559028 8980 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:22:04.562158 8995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:04.563823 8980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:22:04.563977 8980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "8b8faebf9afc4e039d62023404f46902"
format_stamp: "Formatted at 2026-08-07 02:22:04 on dist-test-slave-69rg"
server_key: "02408fdf5266321672afb7cf2d45779f"
server_key_iv: "d81ed5ee9cdcb16e96bf14cd3db23b2a"
server_key_version: "encryptionkey@0"
I20260807 02:22:04.564113 8980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:04.586954 8980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:04.590843 8980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:04.591101 8980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:04.591871 8980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:04.592902 8980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:04.592980 8980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:04.593060 8980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:04.593103 8980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:04.603271 8980 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:38455
I20260807 02:22:04.603283 9111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:38455 every 8 connection(s)
I20260807 02:22:04.604383 8980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:22:04.612469 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 8980
I20260807 02:22:04.612635 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:22:04.613071 4981 external_mini_cluster.cc:1428] Setting key 286aa5f5784c183c58859de5076f5db5
Aug 07 02:22:04 dist-test-slave-69rg krb5kdc[8593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069324, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:04.621443 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:04.606916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:44869 (local address 127.4.221.126:35641)
0807 02:22:04.607144 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:04.607150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:04.608224 (+ 1074us) server_negotiation.cc:407] Connection header received
0807 02:22:04.609899 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:04.609906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:04.609974 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:04.610070 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:04.611630 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.612401 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.614357 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.614652 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.614778 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:04.617642 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:04.617646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:04.617648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:04.617676 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:04.619207 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:04.619794 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:04.619797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:04.619799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:04.619886 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:04.620245 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:04.620250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:04.620252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:04.620459 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:04.620571 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:04.621109 (+ 538us) server_negotiation.cc:299] Negotiation successful
0807 02:22:04.621250 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260807 02:22:04.622494 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:04.607382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:35641 (local address 127.4.221.67:44869)
0807 02:22:04.608004 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:04.608058 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:04.609558 (+ 1500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:04.610271 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:04.610279 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:04.610761 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:04.611433 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:04.611455 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.612608 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.612612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:04.614051 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:04.614068 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.614845 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.614850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:04.615059 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:04.615797 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:04.615815 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:04.617454 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:04.619347 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:04.619354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:04.619373 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:04.619667 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:04.619985 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:04.619988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:04.619990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:04.620145 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:04.620567 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:04.620574 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:04.620857 (+ 283us) client_negotiation.cc:769] Sending connection context
0807 02:22:04.621067 (+ 210us) client_negotiation.cc:240] Negotiation successful
0807 02:22:04.621441 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":168,"threads_started":1}
I20260807 02:22:04.623714 9112 heartbeater.cc:347] Connected to a master server at 127.4.221.126:35641
I20260807 02:22:04.623996 9112 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:04.624571 9112 heartbeater.cc:511] Master 127.4.221.126:35641 requested a full tablet report, sending...
I20260807 02:22:04.625734 8638 ts_manager.cc:194] Registered new tserver with Master: 8b8faebf9afc4e039d62023404f46902 (127.4.221.67:38455)
I20260807 02:22:04.626446 8638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:44869
I20260807 02:22:04.627779 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 02:22:04 dist-test-slave-69rg krb5kdc[8593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 02:22:04 dist-test-slave-69rg krb5kdc[8593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.65@KRBTEST.COM
I20260807 02:22:04.657073 9119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:04.647296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42880 (local address 127.4.221.65:40819)
0807 02:22:04.647688 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:04.647694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:04.647719 (+ 25us) server_negotiation.cc:407] Connection header received
0807 02:22:04.647783 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:04.647788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:04.647995 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:04.648126 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:04.648900 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.649679 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.650629 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.651014 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.651098 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:04.653740 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:04.653751 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:04.653766 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:04.653789 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:04.655392 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:04.655797 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:04.655802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:04.655803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:04.655881 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:04.656110 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:04.656113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:04.656114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:04.656412 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:04.656632 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:04.656757 (+ 125us) server_negotiation.cc:299] Negotiation successful
0807 02:22:04.656918 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":173,"threads_started":1}
Aug 07 02:22:04 dist-test-slave-69rg krb5kdc[8593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:04.669034 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:04.660299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.4.221.126:35641)
0807 02:22:04.660449 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:04.660453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:04.660467 (+ 14us) server_negotiation.cc:407] Connection header received
0807 02:22:04.660566 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:04.660568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:04.660611 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:04.660676 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:04.661274 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.661785 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.662657 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:04.662904 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:04.663021 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:04.665613 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:04.665616 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:04.665618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:04.665647 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:04.667134 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:04.667585 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:04.667588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:04.667593 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:04.667679 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:04.667967 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:04.667970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:04.667972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:04.668258 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:04.668408 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:04.668660 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 02:22:04.668795 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260807 02:22:04.669543 8638 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48870
I20260807 02:22:04.670297 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 8703
I20260807 02:22:04.677893 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 8841
I20260807 02:22:04.685496 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 8980
I20260807 02:22:04.692165 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 8606
2026-08-07T02:22:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2385 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:22:04 dist-test-slave-69rg krb5kdc[9124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:22:04 dist-test-slave-69rg krb5kdc[9124](info): set up 2 sockets
Aug 07 02:22:04 dist-test-slave-69rg krb5kdc[9124](info): commencing operation
krb5kdc: starting...
W20260807 02:22:05.758375 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:22:05 dist-test-slave-69rg krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:22:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:22:05.898613 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:40697
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:40697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:06.027146 9138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:06.027529 9138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:06.027626 9138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:06.031934 9138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:06.032064 9138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:06.032104 9138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:06.032155 9138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:06.032204 9138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:22:06.037981 9138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:40697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:40697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:06.039354 9138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:06.040551 9138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:06.049773 9143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:06.049907 9138 server_base.cc:1034] running on GCE node
W20260807 02:22:06.049746 9144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:06.050112 9146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:06.050798 9138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:06.052146 9138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:06.053381 9138 hybrid_clock.cc:648] HybridClock initialized: now 1786069326053360 us; error 37 us; skew 500 ppm
Aug 07 02:22:06 dist-test-slave-69rg krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069326, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:06.057017 9138 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:22:06.058207 9138 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:06.058307 9138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:06.058570 9138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:06.061944 9138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "5f64552d83c2432bb526a97e37d27774"
format_stamp: "Formatted at 2026-08-07 02:22:06 on dist-test-slave-69rg"
server_key: "1aac9756aa2ce448c8dfd90ead3c71ff"
server_key_iv: "037fe08f2f026f3fef6d46ffc331c40f"
server_key_version: "encryptionkey@0"
I20260807 02:22:06.062834 9138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "5f64552d83c2432bb526a97e37d27774"
format_stamp: "Formatted at 2026-08-07 02:22:06 on dist-test-slave-69rg"
server_key: "1aac9756aa2ce448c8dfd90ead3c71ff"
server_key_iv: "037fe08f2f026f3fef6d46ffc331c40f"
server_key_version: "encryptionkey@0"
I20260807 02:22:06.068778 9138 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260807 02:22:06.074267 9153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:06.076026 9138 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 02:22:06.076308 9138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "5f64552d83c2432bb526a97e37d27774"
format_stamp: "Formatted at 2026-08-07 02:22:06 on dist-test-slave-69rg"
server_key: "1aac9756aa2ce448c8dfd90ead3c71ff"
server_key_iv: "037fe08f2f026f3fef6d46ffc331c40f"
server_key_version: "encryptionkey@0"
I20260807 02:22:06.076480 9138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:06.107484 9138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:06.110690 9138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:06.110913 9138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:06.119258 9138 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:40697
I20260807 02:22:06.119602 9207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:40697 every 8 connection(s)
I20260807 02:22:06.120029 9138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:06.123376 9208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:06.127938 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 9138
I20260807 02:22:06.128098 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:22:06.128409 4981 external_mini_cluster.cc:1428] Setting key 3086bd7c8006ce62e2f5f32487165bd5
I20260807 02:22:06.130502 9208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774: Bootstrap starting.
I20260807 02:22:06.133368 9208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:06.134311 9208 log.cc:824] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:06.136880 9208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774: No bootstrap required, opened a new log
Aug 07 02:22:06 dist-test-slave-69rg krb5kdc[9124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:06.140878 9208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64552d83c2432bb526a97e37d27774" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40697 } }
I20260807 02:22:06.141212 9208 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:06.141302 9208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f64552d83c2432bb526a97e37d27774, State: Initialized, Role: FOLLOWER
I20260807 02:22:06.141804 9208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64552d83c2432bb526a97e37d27774" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40697 } }
I20260807 02:22:06.142017 9208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:06.142110 9208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:06.142247 9208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:06.143257 9208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64552d83c2432bb526a97e37d27774" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40697 } }
I20260807 02:22:06.143654 9208 leader_election.cc:302] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f64552d83c2432bb526a97e37d27774; no voters:
I20260807 02:22:06.144073 9208 leader_election.cc:288] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:22:06.144157 9214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:06.144374 9214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [term 1 LEADER]: Becoming Leader. State: Replica: 5f64552d83c2432bb526a97e37d27774, State: Running, Role: LEADER
I20260807 02:22:06.144721 9214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64552d83c2432bb526a97e37d27774" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40697 } }
I20260807 02:22:06.145563 9208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:06.146476 9216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f64552d83c2432bb526a97e37d27774. Latest consensus state: current_term: 1 leader_uuid: "5f64552d83c2432bb526a97e37d27774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64552d83c2432bb526a97e37d27774" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40697 } } }
I20260807 02:22:06.146924 9216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:06.146806 9215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f64552d83c2432bb526a97e37d27774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64552d83c2432bb526a97e37d27774" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40697 } } }
I20260807 02:22:06.147264 9215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:06.147574 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:06.130113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.4.221.126:40697)
0807 02:22:06.130574 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:06.130599 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:06.130647 (+ 48us) server_negotiation.cc:407] Connection header received
0807 02:22:06.131386 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:06.131407 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:06.131787 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:06.132089 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:06.133072 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:06.134318 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:06.135451 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:06.135935 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:06.136058 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:06.139091 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:06.139096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:06.139112 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:06.139144 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:06.143293 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:06.143827 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:06.143834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:06.143840 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:06.143975 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:06.144333 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:06.144338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:06.144340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:06.145030 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:06.145254 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:06.146002 (+ 748us) server_negotiation.cc:299] Negotiation successful
0807 02:22:06.146394 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":153,"threads_started":1}
I20260807 02:22:06.148417 9223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:06.151911 9223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:06.158541 9223 catalog_manager.cc:1347] Generated new cluster ID: 5c09bfef57494340bdb7205f48a85651
I20260807 02:22:06.158645 9223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:06.170162 9223 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:22:06.171254 9223 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:06.179824 9223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774: Generated new TSK 0
I20260807 02:22:06.180645 9223 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:22:06.222757 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40697
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:06.380427 9235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:06.380752 9235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:06.380864 9235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:06.385196 9235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:06.385281 9235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:06.385349 9235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:22:06.390892 9235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:06.392097 9235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:06.393124 9235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:06.402943 9241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:06.402933 9240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:06.402932 9243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:06.403153 9235 server_base.cc:1034] running on GCE node
I20260807 02:22:06.403717 9235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:06.404515 9235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:06.405710 9235 hybrid_clock.cc:648] HybridClock initialized: now 1786069326405697 us; error 30 us; skew 500 ppm
Aug 07 02:22:06 dist-test-slave-69rg krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069326, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:06.414098 9235 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:22:06.415042 9235 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:06.415115 9235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:06.415398 9235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:06.417603 9235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2"
format_stamp: "Formatted at 2026-08-07 02:22:06 on dist-test-slave-69rg"
server_key: "241680ef38bb322043fd04c2635ae8ce"
server_key_iv: "99f0c7691d32a4fd8df333a41b822f3f"
server_key_version: "encryptionkey@0"
I20260807 02:22:06.418324 9235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2"
format_stamp: "Formatted at 2026-08-07 02:22:06 on dist-test-slave-69rg"
server_key: "241680ef38bb322043fd04c2635ae8ce"
server_key_iv: "99f0c7691d32a4fd8df333a41b822f3f"
server_key_version: "encryptionkey@0"
I20260807 02:22:06.422492 9235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 02:22:06.425206 9249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:06.426610 9235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:22:06.426750 9235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2"
format_stamp: "Formatted at 2026-08-07 02:22:06 on dist-test-slave-69rg"
server_key: "241680ef38bb322043fd04c2635ae8ce"
server_key_iv: "99f0c7691d32a4fd8df333a41b822f3f"
server_key_version: "encryptionkey@0"
I20260807 02:22:06.426878 9235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:06.469286 9235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:06.473014 9235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:06.473268 9235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:06.473954 9235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:06.474855 9235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:06.474905 9235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:06.474947 9235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:06.475006 9235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:06.485301 9235 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:33821
I20260807 02:22:06.485328 9364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:33821 every 8 connection(s)
I20260807 02:22:06.486101 9235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:22:06.493001 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 9235
I20260807 02:22:06.493194 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:22:06.493552 4981 external_mini_cluster.cc:1428] Setting key 0e3caac51291180a69d72ee84970c2e4
Aug 07 02:22:06 dist-test-slave-69rg krb5kdc[9124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069326, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:06.503356 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:06.488636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:45447 (local address 127.4.221.126:40697)
0807 02:22:06.488873 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:06.488878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:06.490004 (+ 1126us) server_negotiation.cc:407] Connection header received
0807 02:22:06.491576 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:06.491579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:06.491649 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:06.491738 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:06.493256 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:06.494094 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:06.495745 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:06.496141 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:06.496340 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:06.499230 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:06.499233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:06.499236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:06.499262 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:06.500814 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:06.501537 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:06.501539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:06.501540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:06.501619 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:06.502091 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:06.502094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:06.502095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:06.502289 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:06.502370 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:06.503109 (+ 739us) server_negotiation.cc:299] Negotiation successful
0807 02:22:06.503225 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260807 02:22:06.504755 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:06.489137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40697 (local address 127.4.221.65:45447)
0807 02:22:06.489733 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:06.489806 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:06.491320 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:06.491963 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:06.491970 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:06.492403 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:06.492977 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:06.492995 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:06.494266 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:06.494269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:06.495596 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:06.495607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:06.496367 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:06.496370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:06.496520 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:06.497193 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:06.497211 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:06.499007 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:06.500949 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:06.500959 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:06.500980 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:06.501424 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:06.501727 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:06.501733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:06.501737 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:06.501987 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:06.502410 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:06.502420 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:06.502828 (+ 408us) client_negotiation.cc:769] Sending connection context
0807 02:22:06.503057 (+ 229us) client_negotiation.cc:240] Negotiation successful
0807 02:22:06.503475 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":158,"threads_started":1}
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
I20260807 02:22:06.506213 9365 heartbeater.cc:347] Connected to a master server at 127.4.221.126:40697
I20260807 02:22:06.506474 9365 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:06.507040 9365 heartbeater.cc:511] Master 127.4.221.126:40697 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
I20260807 02:22:06.508841 9171 ts_manager.cc:194] Registered new tserver with Master: b10d6a9b1e1e48d4b066ba8c21f915b2 (127.4.221.65:33821)
I20260807 02:22:06.510221 9171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:45447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:22:06.530109 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40697
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:06.650748 9370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:06.651098 9370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:06.651185 9370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:06.655644 9370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:06.655781 9370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:06.655887 9370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:22:06.661936 9370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:06.663210 9370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:06.664320 9370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:06.674757 9379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:06.674774 9377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:06.674798 9370 server_base.cc:1034] running on GCE node
W20260807 02:22:06.675112 9376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:06.675664 9370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:06.676537 9370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:06.677744 9370 hybrid_clock.cc:648] HybridClock initialized: now 1786069326677664 us; error 124 us; skew 500 ppm
Aug 07 02:22:06 dist-test-slave-69rg krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069326, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:06.681131 9370 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:22:06.682188 9370 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:06.682260 9370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:06.682490 9370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:06.684700 9370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "ed4ee2da08584c9dac9be5994d96e6aa"
format_stamp: "Formatted at 2026-08-07 02:22:06 on dist-test-slave-69rg"
server_key: "cb3fc017602441dfe066717152c6355b"
server_key_iv: "1873f82f67c46a174256083cd7facfc5"
server_key_version: "encryptionkey@0"
I20260807 02:22:06.685371 9370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "ed4ee2da08584c9dac9be5994d96e6aa"
format_stamp: "Formatted at 2026-08-07 02:22:06 on dist-test-slave-69rg"
server_key: "cb3fc017602441dfe066717152c6355b"
server_key_iv: "1873f82f67c46a174256083cd7facfc5"
server_key_version: "encryptionkey@0"
I20260807 02:22:06.689558 9370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260807 02:22:06.692138 9385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:06.693399 9370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:22:06.693542 9370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "ed4ee2da08584c9dac9be5994d96e6aa"
format_stamp: "Formatted at 2026-08-07 02:22:06 on dist-test-slave-69rg"
server_key: "cb3fc017602441dfe066717152c6355b"
server_key_iv: "1873f82f67c46a174256083cd7facfc5"
server_key_version: "encryptionkey@0"
I20260807 02:22:06.693660 9370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:06.722182 9370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:06.725845 9370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:06.726018 9370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:06.726596 9370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:06.727448 9370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:06.727497 9370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:06.727540 9370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:06.727555 9370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:06.737808 9370 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:42523
I20260807 02:22:06.737850 9502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:42523 every 8 connection(s)
I20260807 02:22:06.738648 9370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:22:06.748689 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 9370
Aug 07 02:22:06 dist-test-slave-69rg krb5kdc[9124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069326, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:06.748809 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:22:06.749181 4981 external_mini_cluster.cc:1428] Setting key e115ea3d4a0e6bf5ca4c5b5b78ec1f71
I20260807 02:22:06.754315 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:06.740527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:44567 (local address 127.4.221.126:40697)
0807 02:22:06.740673 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:06.740677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:06.741464 (+ 787us) server_negotiation.cc:407] Connection header received
0807 02:22:06.742998 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:06.743002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:06.743064 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:06.743145 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:06.744350 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:06.744898 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:06.746258 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:06.746532 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:06.746677 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:06.749726 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:06.749731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:06.749733 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:06.749768 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:06.751530 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:06.752270 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:06.752273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:06.752275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:06.752357 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:06.752840 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:06.752843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:06.752844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:06.753049 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:06.753230 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:06.753902 (+ 672us) server_negotiation.cc:299] Negotiation successful
0807 02:22:06.754074 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 02:22:06.755143 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:06.740853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40697 (local address 127.4.221.66:44567)
0807 02:22:06.741316 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:06.741360 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:06.742773 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:06.743321 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:06.743329 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:06.743697 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:06.744191 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:06.744204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:06.745032 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:06.745035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:06.746050 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:06.746060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:06.746753 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:06.746757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:06.746936 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:06.747610 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:06.747627 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:06.749469 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:06.751670 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:06.751697 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:06.751720 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:06.752154 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:06.752457 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:06.752463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:06.752467 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:06.752702 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:06.753182 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:06.753190 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:06.753594 (+ 404us) client_negotiation.cc:769] Sending connection context
0807 02:22:06.753778 (+ 184us) client_negotiation.cc:240] Negotiation successful
0807 02:22:06.754165 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":120,"threads_started":1}
I20260807 02:22:06.756593 9503 heartbeater.cc:347] Connected to a master server at 127.4.221.126:40697
I20260807 02:22:06.756893 9503 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:06.757544 9503 heartbeater.cc:511] Master 127.4.221.126:40697 requested a full tablet report, sending...
I20260807 02:22:06.758967 9171 ts_manager.cc:194] Registered new tserver with Master: ed4ee2da08584c9dac9be5994d96e6aa (127.4.221.66:42523)
I20260807 02:22:06.759631 9171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:44567
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:22:06.782291 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40697
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:06.900388 9508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:06.900739 9508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:06.900822 9508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:06.905133 9508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:06.905273 9508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:06.905385 9508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:22:06.911566 9508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:06.912899 9508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:06.914063 9508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:06.924153 9515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:06.924142 9517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:06.924142 9514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:06.924353 9508 server_base.cc:1034] running on GCE node
I20260807 02:22:06.924984 9508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:06.925729 9508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:06.926908 9508 hybrid_clock.cc:648] HybridClock initialized: now 1786069326926888 us; error 33 us; skew 500 ppm
Aug 07 02:22:06 dist-test-slave-69rg krb5kdc[9124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069326, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:06.930055 9508 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:22:06.931012 9508 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:06.931118 9508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:06.931430 9508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:06.933622 9508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "49b93167853d40378af17b1e466f7aaf"
format_stamp: "Formatted at 2026-08-07 02:22:06 on dist-test-slave-69rg"
server_key: "3448d51d5da484ecb19e625a20347652"
server_key_iv: "244cf51998e4739a8873ae6023ffbb77"
server_key_version: "encryptionkey@0"
I20260807 02:22:06.934306 9508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "49b93167853d40378af17b1e466f7aaf"
format_stamp: "Formatted at 2026-08-07 02:22:06 on dist-test-slave-69rg"
server_key: "3448d51d5da484ecb19e625a20347652"
server_key_iv: "244cf51998e4739a8873ae6023ffbb77"
server_key_version: "encryptionkey@0"
I20260807 02:22:06.938189 9508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 02:22:06.940891 9524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:06.942408 9508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:22:06.942551 9508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "49b93167853d40378af17b1e466f7aaf"
format_stamp: "Formatted at 2026-08-07 02:22:06 on dist-test-slave-69rg"
server_key: "3448d51d5da484ecb19e625a20347652"
server_key_iv: "244cf51998e4739a8873ae6023ffbb77"
server_key_version: "encryptionkey@0"
I20260807 02:22:06.942675 9508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:06.958020 9508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:06.961480 9508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:06.961712 9508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:06.962436 9508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:06.963351 9508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:06.963402 9508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:06.963470 9508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:06.963506 9508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:06.972995 9508 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:44757
I20260807 02:22:06.973014 9639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:44757 every 8 connection(s)
I20260807 02:22:06.973637 9508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:22:06.980413 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 9508
I20260807 02:22:06.980594 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:22:06.980999 4981 external_mini_cluster.cc:1428] Setting key 1e62ff37778eaec69bb448700a1e5c78
Aug 07 02:22:06 dist-test-slave-69rg krb5kdc[9124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069326, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:06.988209 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:06.975522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:55527 (local address 127.4.221.126:40697)
0807 02:22:06.975667 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:06.975671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:06.976556 (+ 885us) server_negotiation.cc:407] Connection header received
0807 02:22:06.977968 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:06.977971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:06.978032 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:06.978166 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:06.979420 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:06.979978 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:06.981356 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:06.981588 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:06.981713 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:06.984458 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:06.984469 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:06.984472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:06.984499 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:06.985950 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:06.986499 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:06.986503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:06.986505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:06.986594 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:06.986958 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:06.986962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:06.986963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:06.987169 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:06.987281 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:06.987900 (+ 619us) server_negotiation.cc:299] Negotiation successful
0807 02:22:06.988035 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:22:06.989086 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:06.975903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40697 (local address 127.4.221.67:55527)
0807 02:22:06.976389 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:06.976437 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:06.977714 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:06.978359 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:06.978366 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:06.978750 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:06.979253 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:06.979267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:06.980139 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:06.980143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:06.981221 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:06.981233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:06.981765 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:06.981770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:06.981999 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:06.982668 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:06.982686 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:06.984259 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:06.986074 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:06.986081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:06.986098 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:06.986385 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:06.986692 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:06.986695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:06.986697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:06.986850 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:06.987282 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:06.987290 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:06.987643 (+ 353us) client_negotiation.cc:769] Sending connection context
0807 02:22:06.987851 (+ 208us) client_negotiation.cc:240] Negotiation successful
0807 02:22:06.988208 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":126,"threads_started":1}
I20260807 02:22:06.990424 9640 heartbeater.cc:347] Connected to a master server at 127.4.221.126:40697
I20260807 02:22:06.990738 9640 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:06.991263 9640 heartbeater.cc:511] Master 127.4.221.126:40697 requested a full tablet report, sending...
I20260807 02:22:06.992439 9171 ts_manager.cc:194] Registered new tserver with Master: 49b93167853d40378af17b1e466f7aaf (127.4.221.67:44757)
I20260807 02:22:06.993014 9171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:55527
I20260807 02:22:06.995347 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:22:07.004951 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:06.997442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56404 (local address 127.4.221.126:40697)
0807 02:22:06.997574 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:06.997585 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:06.997772 (+ 187us) server_negotiation.cc:407] Connection header received
0807 02:22:06.997971 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:06.997974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:06.998015 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:06.998083 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:06.998905 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:06.999411 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.000250 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.000476 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.000568 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:07.002111 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:07.002115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:07.002117 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:07.002145 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:07.003473 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:07.003880 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:07.003883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:07.003884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:07.003961 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:07.004192 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:07.004194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:07.004195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:07.004429 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:07.004532 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:07.004707 (+ 175us) server_negotiation.cc:299] Negotiation successful
0807 02:22:07.004820 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 02:22:07.007993 9171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:22:07.011108 9171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:22:07.026222 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.020955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47024 (local address 127.4.221.66:42523)
0807 02:22:07.021373 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:07.021388 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:07.021621 (+ 233us) server_negotiation.cc:407] Connection header received
0807 02:22:07.021757 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:07.021767 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:07.021971 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:07.022118 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:07.022330 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.023095 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.024650 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.025406 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.025520 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:07.025682 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:07.025935 (+ 253us) server_negotiation.cc:299] Negotiation successful
0807 02:22:07.026049 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":96,"threads_started":1}
I20260807 02:22:07.026218 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.020667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:42523 (local address 127.0.0.1:47024)
0807 02:22:07.021513 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:07.021524 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:07.021645 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:07.022098 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:07.022101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:07.022119 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:07.022232 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:07.022241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.023226 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.023231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:07.024527 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:07.024537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.025545 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.025549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:07.025659 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:07.025703 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 02:22:07.025886 (+ 183us) client_negotiation.cc:240] Negotiation successful
0807 02:22:07.026044 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
I20260807 02:22:07.026665 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.019964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:44757 (local address 127.0.0.1:51206)
0807 02:22:07.020922 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:07.020981 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:07.021133 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:07.021887 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:07.021890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:07.021927 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:07.022158 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:07.022165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.023652 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.023656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:07.024876 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:07.024887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.026178 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.026181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:07.026305 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:07.026358 (+ 53us) client_negotiation.cc:769] Sending connection context
0807 02:22:07.026433 (+ 75us) client_negotiation.cc:240] Negotiation successful
0807 02:22:07.026518 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":82,"spinlock_wait_cycles":6656,"thread_start_us":127,"threads_started":1}
I20260807 02:22:07.027127 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.019890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33821 (local address 127.0.0.1:36012)
0807 02:22:07.020923 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:07.020980 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:07.021131 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:07.021884 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:07.021890 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:07.021926 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:07.022158 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:07.022165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.023652 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.023655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:07.025057 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:07.025064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.026254 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.026260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:07.026696 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:07.026722 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 02:22:07.026926 (+ 204us) client_negotiation.cc:240] Negotiation successful
0807 02:22:07.027000 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":516,"spinlock_wait_cycles":6656,"thread_start_us":107,"threads_started":1}
I20260807 02:22:07.027179 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.020193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51206 (local address 127.4.221.67:44757)
0807 02:22:07.020615 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:07.020631 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:07.021077 (+ 446us) server_negotiation.cc:407] Connection header received
0807 02:22:07.021326 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:07.021332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:07.021545 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:07.021671 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:07.022309 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.023504 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.025009 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.025881 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.025988 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:07.026111 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:07.026909 (+ 798us) server_negotiation.cc:299] Negotiation successful
0807 02:22:07.027018 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":122,"threads_started":1}
I20260807 02:22:07.027649 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.020006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36012 (local address 127.4.221.65:33821)
0807 02:22:07.020788 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:07.020795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:07.021275 (+ 480us) server_negotiation.cc:407] Connection header received
0807 02:22:07.021351 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:07.021355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:07.021545 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:07.021674 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:07.022300 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.023504 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.025182 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.025954 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.026054 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:07.026186 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:07.027384 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0807 02:22:07.027504 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":77,"threads_started":1}
I20260807 02:22:07.030197 9573 tablet_service.cc:1467] Processing CreateTablet for tablet 66de19eacf964f50b1485a3a881ab646 (DEFAULT_TABLE table=test-table [id=9ea63b12fbf44302aacb3c0a2c4260bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:07.030202 9297 tablet_service.cc:1467] Processing CreateTablet for tablet 66de19eacf964f50b1485a3a881ab646 (DEFAULT_TABLE table=test-table [id=9ea63b12fbf44302aacb3c0a2c4260bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:07.030205 9435 tablet_service.cc:1467] Processing CreateTablet for tablet 66de19eacf964f50b1485a3a881ab646 (DEFAULT_TABLE table=test-table [id=9ea63b12fbf44302aacb3c0a2c4260bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:07.031348 9435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66de19eacf964f50b1485a3a881ab646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:07.031348 9297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66de19eacf964f50b1485a3a881ab646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:07.031375 9573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66de19eacf964f50b1485a3a881ab646. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:07.037923 9658 tablet_bootstrap.cc:492] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf: Bootstrap starting.
I20260807 02:22:07.039366 9660 tablet_bootstrap.cc:492] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa: Bootstrap starting.
I20260807 02:22:07.040208 9658 tablet_bootstrap.cc:654] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:07.041067 9660 tablet_bootstrap.cc:654] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:07.041368 9658 log.cc:824] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:07.042146 9659 tablet_bootstrap.cc:492] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2: Bootstrap starting.
I20260807 02:22:07.042284 9660 log.cc:824] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:07.043843 9658 tablet_bootstrap.cc:492] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf: No bootstrap required, opened a new log
I20260807 02:22:07.044034 9660 tablet_bootstrap.cc:492] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa: No bootstrap required, opened a new log
I20260807 02:22:07.044050 9658 ts_tablet_manager.cc:1397] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 02:22:07.044230 9660 ts_tablet_manager.cc:1397] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 02:22:07.044490 9659 tablet_bootstrap.cc:654] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:07.045382 9659 log.cc:824] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:07.047741 9659 tablet_bootstrap.cc:492] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2: No bootstrap required, opened a new log
I20260807 02:22:07.048056 9659 ts_tablet_manager.cc:1397] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 02:22:07.048153 9658 raft_consensus.cc:348] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33821 } } peers { permanent_uuid: "49b93167853d40378af17b1e466f7aaf" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44757 } } peers { permanent_uuid: "ed4ee2da08584c9dac9be5994d96e6aa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 42523 } }
I20260807 02:22:07.048462 9658 raft_consensus.cc:374] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:07.048295 9660 raft_consensus.cc:348] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33821 } } peers { permanent_uuid: "49b93167853d40378af17b1e466f7aaf" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44757 } } peers { permanent_uuid: "ed4ee2da08584c9dac9be5994d96e6aa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 42523 } }
I20260807 02:22:07.048534 9658 raft_consensus.cc:729] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49b93167853d40378af17b1e466f7aaf, State: Initialized, Role: FOLLOWER
I20260807 02:22:07.048574 9660 raft_consensus.cc:374] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:07.048641 9660 raft_consensus.cc:729] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4ee2da08584c9dac9be5994d96e6aa, State: Initialized, Role: FOLLOWER
I20260807 02:22:07.049007 9658 consensus_queue.cc:260] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33821 } } peers { permanent_uuid: "49b93167853d40378af17b1e466f7aaf" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44757 } } peers { permanent_uuid: "ed4ee2da08584c9dac9be5994d96e6aa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 42523 } }
I20260807 02:22:07.049392 9660 consensus_queue.cc:260] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33821 } } peers { permanent_uuid: "49b93167853d40378af17b1e466f7aaf" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44757 } } peers { permanent_uuid: "ed4ee2da08584c9dac9be5994d96e6aa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 42523 } }
I20260807 02:22:07.050438 9660 ts_tablet_manager.cc:1428] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 02:22:07.050540 9658 ts_tablet_manager.cc:1428] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 02:22:07.050894 9503 heartbeater.cc:503] Master 127.4.221.126:40697 was elected leader, sending a full tablet report...
I20260807 02:22:07.050915 9640 heartbeater.cc:503] Master 127.4.221.126:40697 was elected leader, sending a full tablet report...
I20260807 02:22:07.052970 9659 raft_consensus.cc:348] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33821 } } peers { permanent_uuid: "49b93167853d40378af17b1e466f7aaf" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44757 } } peers { permanent_uuid: "ed4ee2da08584c9dac9be5994d96e6aa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 42523 } }
I20260807 02:22:07.053313 9659 raft_consensus.cc:374] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:07.053395 9659 raft_consensus.cc:729] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b10d6a9b1e1e48d4b066ba8c21f915b2, State: Initialized, Role: FOLLOWER
I20260807 02:22:07.054039 9659 consensus_queue.cc:260] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33821 } } peers { permanent_uuid: "49b93167853d40378af17b1e466f7aaf" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44757 } } peers { permanent_uuid: "ed4ee2da08584c9dac9be5994d96e6aa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 42523 } }
I20260807 02:22:07.055107 9365 heartbeater.cc:503] Master 127.4.221.126:40697 was elected leader, sending a full tablet report...
I20260807 02:22:07.055559 9659 ts_tablet_manager.cc:1428] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 02:22:07.064544 9666 raft_consensus.cc:482] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:22:07.064810 9666 raft_consensus.cc:504] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33821 } } peers { permanent_uuid: "49b93167853d40378af17b1e466f7aaf" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44757 } } peers { permanent_uuid: "ed4ee2da08584c9dac9be5994d96e6aa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 42523 } }
I20260807 02:22:07.066130 9666 leader_election.cc:288] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49b93167853d40378af17b1e466f7aaf (127.4.221.67:44757), ed4ee2da08584c9dac9be5994d96e6aa (127.4.221.66:42523)
I20260807 02:22:07.070871 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.066541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:37385 (local address 127.4.221.66:42523)
0807 02:22:07.066672 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:07.066677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:07.066894 (+ 217us) server_negotiation.cc:407] Connection header received
0807 02:22:07.067048 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:07.067054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:07.067102 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:07.067184 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:07.067480 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.068046 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.069515 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.070182 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.070337 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:07.070394 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:07.070655 (+ 261us) server_negotiation.cc:299] Negotiation successful
0807 02:22:07.070730 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 02:22:07.070863 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.066477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:58063 (local address 127.4.221.67:44757)
0807 02:22:07.066616 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:07.066621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:07.066951 (+ 330us) server_negotiation.cc:407] Connection header received
0807 02:22:07.067096 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:07.067099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:07.067150 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:07.067233 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:07.067534 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.068054 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.069573 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.070260 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.070367 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:07.070410 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:07.070655 (+ 245us) server_negotiation.cc:299] Negotiation successful
0807 02:22:07.070725 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 02:22:07.070909 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.066404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:42523 (local address 127.4.221.65:37385)
0807 02:22:07.066794 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:07.066807 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:07.066947 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:07.067206 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:07.067209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:07.067224 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:07.067370 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:07.067376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.068174 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.068177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:07.069353 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:07.069363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.070326 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.070332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:07.070440 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:07.070535 (+ 95us) client_negotiation.cc:769] Sending connection context
0807 02:22:07.070619 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 02:22:07.070718 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":59,"thread_start_us":133,"threads_started":1}
I20260807 02:22:07.071156 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.066408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:44757 (local address 127.4.221.65:58063)
0807 02:22:07.066862 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:07.066875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:07.066973 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:07.067286 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:07.067289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:07.067305 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:07.067415 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:07.067420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.068218 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.068222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:07.069399 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:07.069412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.070360 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.070363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:07.070477 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:07.070538 (+ 61us) client_negotiation.cc:769] Sending connection context
0807 02:22:07.070623 (+ 85us) client_negotiation.cc:240] Negotiation successful
0807 02:22:07.070717 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":86,"thread_start_us":105,"threads_started":1}
I20260807 02:22:07.071578 9455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66de19eacf964f50b1485a3a881ab646" candidate_uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4ee2da08584c9dac9be5994d96e6aa" is_pre_election: true
I20260807 02:22:07.071799 9593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66de19eacf964f50b1485a3a881ab646" candidate_uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49b93167853d40378af17b1e466f7aaf" is_pre_election: true
I20260807 02:22:07.071908 9455 raft_consensus.cc:2454] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b10d6a9b1e1e48d4b066ba8c21f915b2 in term 0.
I20260807 02:22:07.072098 9593 raft_consensus.cc:2454] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b10d6a9b1e1e48d4b066ba8c21f915b2 in term 0.
I20260807 02:22:07.072420 9253 leader_election.cc:302] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b10d6a9b1e1e48d4b066ba8c21f915b2, ed4ee2da08584c9dac9be5994d96e6aa; no voters:
I20260807 02:22:07.072749 9666 raft_consensus.cc:2781] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:22:07.072827 9666 raft_consensus.cc:482] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:22:07.072858 9666 raft_consensus.cc:3037] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:07.073999 9666 raft_consensus.cc:504] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33821 } } peers { permanent_uuid: "49b93167853d40378af17b1e466f7aaf" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44757 } } peers { permanent_uuid: "ed4ee2da08584c9dac9be5994d96e6aa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 42523 } }
I20260807 02:22:07.074383 9666 leader_election.cc:288] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [CANDIDATE]: Term 1 election: Requested vote from peers 49b93167853d40378af17b1e466f7aaf (127.4.221.67:44757), ed4ee2da08584c9dac9be5994d96e6aa (127.4.221.66:42523)
I20260807 02:22:07.074728 9593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66de19eacf964f50b1485a3a881ab646" candidate_uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49b93167853d40378af17b1e466f7aaf"
I20260807 02:22:07.074787 9455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66de19eacf964f50b1485a3a881ab646" candidate_uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4ee2da08584c9dac9be5994d96e6aa"
I20260807 02:22:07.074849 9593 raft_consensus.cc:3037] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:07.074986 9455 raft_consensus.cc:3037] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:07.075960 9593 raft_consensus.cc:2454] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b10d6a9b1e1e48d4b066ba8c21f915b2 in term 1.
I20260807 02:22:07.076148 9455 raft_consensus.cc:2454] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b10d6a9b1e1e48d4b066ba8c21f915b2 in term 1.
I20260807 02:22:07.076339 9250 leader_election.cc:302] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49b93167853d40378af17b1e466f7aaf, b10d6a9b1e1e48d4b066ba8c21f915b2; no voters:
I20260807 02:22:07.076582 9666 raft_consensus.cc:2781] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:07.076833 9666 raft_consensus.cc:686] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [term 1 LEADER]: Becoming Leader. State: Replica: b10d6a9b1e1e48d4b066ba8c21f915b2, State: Running, Role: LEADER
I20260807 02:22:07.077160 9666 consensus_queue.cc:237] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33821 } } peers { permanent_uuid: "49b93167853d40378af17b1e466f7aaf" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44757 } } peers { permanent_uuid: "ed4ee2da08584c9dac9be5994d96e6aa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 42523 } }
I20260807 02:22:07.080673 9170 catalog_manager.cc:5495] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b10d6a9b1e1e48d4b066ba8c21f915b2 (127.4.221.65). New cstate: current_term: 1 leader_uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b10d6a9b1e1e48d4b066ba8c21f915b2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49b93167853d40378af17b1e466f7aaf" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed4ee2da08584c9dac9be5994d96e6aa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 42523 } health_report { overall_health: UNKNOWN } } }
I20260807 02:22:07.106844 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.103188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.4.221.65:33821)
0807 02:22:07.103333 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:07.103337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:07.103367 (+ 30us) server_negotiation.cc:407] Connection header received
0807 02:22:07.103448 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:07.103450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:07.103511 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:07.103595 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:22:07.103830 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.104371 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.105228 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:07.105462 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:07.105558 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:07.105978 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:22:07.106475 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:22:07.106578 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:07.106652 (+ 74us) server_negotiation.cc:299] Negotiation successful
0807 02:22:07.106721 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:22:07.113070 9593 raft_consensus.cc:1261] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf [term 1 FOLLOWER]: Refusing update from remote peer b10d6a9b1e1e48d4b066ba8c21f915b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:07.113070 9455 raft_consensus.cc:1261] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa [term 1 FOLLOWER]: Refusing update from remote peer b10d6a9b1e1e48d4b066ba8c21f915b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:07.113776 9666 consensus_queue.cc:1035] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed4ee2da08584c9dac9be5994d96e6aa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 42523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:07.114044 9669 consensus_queue.cc:1035] T 66de19eacf964f50b1485a3a881ab646 P b10d6a9b1e1e48d4b066ba8c21f915b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49b93167853d40378af17b1e466f7aaf" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:07.126274 9675 mvcc.cc:205] Tried to move back new op lower bound from 7315739963847499776 to 7315739963710963712. Current Snapshot: MvccSnapshot[applied={T|T < 7315739963847499776}]
I20260807 02:22:07.132663 9169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:07.132897 9169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:07.135752 9169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f64552d83c2432bb526a97e37d27774: Sending DeleteTablet for 3 replicas of tablet 66de19eacf964f50b1485a3a881ab646
I20260807 02:22:07.136643 9297 tablet_service.cc:1514] Processing DeleteTablet for tablet 66de19eacf964f50b1485a3a881ab646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:07 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:36012
I20260807 02:22:07.136974 9435 tablet_service.cc:1514] Processing DeleteTablet for tablet 66de19eacf964f50b1485a3a881ab646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:07 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:47024
I20260807 02:22:07.137255 9573 tablet_service.cc:1514] Processing DeleteTablet for tablet 66de19eacf964f50b1485a3a881ab646 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:07 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:51206
I20260807 02:22:07.137621 9685 tablet_replica.cc:330] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa: stopping tablet replica
I20260807 02:22:07.137743 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 9235
I20260807 02:22:07.139426 9685 raft_consensus.cc:2229] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:07.139814 9685 raft_consensus.cc:2258] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:07.140662 9686 tablet_replica.cc:330] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf: stopping tablet replica
I20260807 02:22:07.140986 9686 raft_consensus.cc:2229] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:07.141283 9686 raft_consensus.cc:2258] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:07.141459 9685 ts_tablet_manager.cc:1905] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:07.142768 9686 ts_tablet_manager.cc:1905] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:07.144968 9686 ts_tablet_manager.cc:1918] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:07.145089 9686 log.cc:1196] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/66de19eacf964f50b1485a3a881ab646
I20260807 02:22:07.145259 9685 ts_tablet_manager.cc:1918] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:07.145375 9685 log.cc:1196] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/66de19eacf964f50b1485a3a881ab646
I20260807 02:22:07.145433 9686 ts_tablet_manager.cc:1939] T 66de19eacf964f50b1485a3a881ab646 P 49b93167853d40378af17b1e466f7aaf: Deleting consensus metadata
I20260807 02:22:07.145748 9685 ts_tablet_manager.cc:1939] T 66de19eacf964f50b1485a3a881ab646 P ed4ee2da08584c9dac9be5994d96e6aa: Deleting consensus metadata
I20260807 02:22:07.147102 9154 catalog_manager.cc:4841] TS 49b93167853d40378af17b1e466f7aaf (127.4.221.67:44757): tablet 66de19eacf964f50b1485a3a881ab646 (table test-table [id=9ea63b12fbf44302aacb3c0a2c4260bc]) successfully deleted
W20260807 02:22:07.147746 9155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:33821 (error 108)
I20260807 02:22:07.148011 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 9370
I20260807 02:22:07.149041 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.148688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33821 (local address 127.0.0.1:36040)
0807 02:22:07.148845 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:07.148933 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 02:22:07.149204 9155 catalog_manager.cc:4584] TS b10d6a9b1e1e48d4b066ba8c21f915b2 (127.4.221.65:33821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 66de19eacf964f50b1485a3a881ab646: Network error: Client connection negotiation failed: client connection to 127.4.221.65:33821: connect: Connection refused (error 111)
I20260807 02:22:07.150720 9157 catalog_manager.cc:4841] TS ed4ee2da08584c9dac9be5994d96e6aa (127.4.221.66:42523): tablet 66de19eacf964f50b1485a3a881ab646 (table test-table [id=9ea63b12fbf44302aacb3c0a2c4260bc]) successfully deleted
I20260807 02:22:07.154877 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 9508
I20260807 02:22:07.161747 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 9138
2026-08-07T02:22:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2469 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-07T02:22:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:07Z Disabled control of system clock
I20260807 02:22:07.196321 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:34959
--webserver_interface=127.4.221.126
--webserver_port=0
--builtin_ntp_servers=127.4.221.84:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:34959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 02:22:07.314254 9691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:07.314574 9691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:07.314615 9691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:07.318708 9691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:07.318773 9691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:07.318789 9691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:07.318807 9691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:07.318822 9691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:22:07.323781 9691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:34959
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:34959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:07.324914 9691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:07.326349 9691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:07.335354 9701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:07.335399 9698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:07.335354 9699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:07.335425 9691 server_base.cc:1034] running on GCE node
I20260807 02:22:07.336212 9691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:07.337339 9691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:07.338511 9691 hybrid_clock.cc:648] HybridClock initialized: now 1786069327338482 us; error 41 us; skew 500 ppm
I20260807 02:22:07.340561 9691 webserver.cc:460] Webserver started at http://127.4.221.126:41005/ using document root <none> and password file <none>
I20260807 02:22:07.341172 9691 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:07.341238 9691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:07.341507 9691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:07.343708 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "7e095761372f45a98d6f670292031b64"
format_stamp: "Formatted at 2026-08-07 02:22:07 on dist-test-slave-69rg"
server_key: "60e0d9f12898c92d18f5482c7b9311aa"
server_key_iv: "bce11101abd97b8d6d207fafeffe311b"
server_key_version: "encryptionkey@0"
I20260807 02:22:07.344333 9691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "7e095761372f45a98d6f670292031b64"
format_stamp: "Formatted at 2026-08-07 02:22:07 on dist-test-slave-69rg"
server_key: "60e0d9f12898c92d18f5482c7b9311aa"
server_key_iv: "bce11101abd97b8d6d207fafeffe311b"
server_key_version: "encryptionkey@0"
I20260807 02:22:07.348489 9691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 02:22:07.351348 9707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:07.352720 9691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:22:07.352844 9691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "7e095761372f45a98d6f670292031b64"
format_stamp: "Formatted at 2026-08-07 02:22:07 on dist-test-slave-69rg"
server_key: "60e0d9f12898c92d18f5482c7b9311aa"
server_key_iv: "bce11101abd97b8d6d207fafeffe311b"
server_key_version: "encryptionkey@0"
I20260807 02:22:07.352975 9691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:07.358584 9691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:07.359251 9691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:07.359465 9691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:07.367147 9691 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:34959
I20260807 02:22:07.367165 9760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:34959 every 8 connection(s)
I20260807 02:22:07.368237 9691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:07.372087 9761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:07.373538 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 9691
I20260807 02:22:07.373680 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:22:07.374044 4981 external_mini_cluster.cc:1428] Setting key 4acaf3db02b2e30732df620651b93b80
I20260807 02:22:07.379412 9761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64: Bootstrap starting.
I20260807 02:22:07.380162 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.375634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.4.221.126:34959)
0807 02:22:07.376376 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:07.376395 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:07.376432 (+ 37us) server_negotiation.cc:407] Connection header received
0807 02:22:07.377124 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:07.377143 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:07.377524 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:07.377913 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:07.378792 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:07.378797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:07.378809 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:07.378821 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:07.378885 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:07.378902 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:07.379073 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:07.379151 (+ 78us) server_negotiation.cc:299] Negotiation successful
0807 02:22:07.379401 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":152,"threads_started":1}
I20260807 02:22:07.382123 9761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:07.383033 9761 log.cc:824] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:07.385228 9761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64: No bootstrap required, opened a new log
I20260807 02:22:07.389314 9761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e095761372f45a98d6f670292031b64" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34959 } }
I20260807 02:22:07.389617 9761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:07.389707 9761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e095761372f45a98d6f670292031b64, State: Initialized, Role: FOLLOWER
I20260807 02:22:07.390239 9761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e095761372f45a98d6f670292031b64" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34959 } }
I20260807 02:22:07.390403 9761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:07.390497 9761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:07.390609 9761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:07.391692 9761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e095761372f45a98d6f670292031b64" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34959 } }
I20260807 02:22:07.392072 9761 leader_election.cc:302] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e095761372f45a98d6f670292031b64; no voters:
I20260807 02:22:07.392438 9761 leader_election.cc:288] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:22:07.392599 9766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:07.392867 9766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [term 1 LEADER]: Becoming Leader. State: Replica: 7e095761372f45a98d6f670292031b64, State: Running, Role: LEADER
I20260807 02:22:07.393260 9766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e095761372f45a98d6f670292031b64" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34959 } }
I20260807 02:22:07.394039 9761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:07.395222 9767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e095761372f45a98d6f670292031b64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e095761372f45a98d6f670292031b64" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34959 } } }
I20260807 02:22:07.395222 9768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e095761372f45a98d6f670292031b64. Latest consensus state: current_term: 1 leader_uuid: "7e095761372f45a98d6f670292031b64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e095761372f45a98d6f670292031b64" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34959 } } }
I20260807 02:22:07.395411 9767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:07.395473 9768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:07.395937 9776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:07.399279 9776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:07.405309 9776 catalog_manager.cc:1347] Generated new cluster ID: f2fac5024ca8485e8f565da4e417c8fa
I20260807 02:22:07.405411 9776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:07.441182 9776 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:22:07.441601 9776 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:07.453399 9776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64: Generated new TSK 0
I20260807 02:22:07.454350 9776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:22:07.464051 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:34959
--builtin_ntp_servers=127.4.221.84:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 02:22:07.581781 9786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:07.582156 9786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:07.582203 9786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:07.586714 9786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:07.586809 9786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:07.586880 9786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:22:07.592139 9786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:34959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:07.593298 9786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:07.594815 9786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:07.604648 9792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:07.604746 9795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:07.604713 9793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:07.605031 9786 server_base.cc:1034] running on GCE node
I20260807 02:22:07.605453 9786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:07.606187 9786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:07.607391 9786 hybrid_clock.cc:648] HybridClock initialized: now 1786069327607354 us; error 46 us; skew 500 ppm
I20260807 02:22:07.609316 9786 webserver.cc:460] Webserver started at http://127.4.221.65:44743/ using document root <none> and password file <none>
I20260807 02:22:07.610008 9786 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:07.610110 9786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:07.610358 9786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:07.612527 9786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "898a3f423ddb4d17b0ba433b35fa1b40"
format_stamp: "Formatted at 2026-08-07 02:22:07 on dist-test-slave-69rg"
server_key: "966d9434058b8584296025c29e91274c"
server_key_iv: "bd455bd57827a4543627c0b720164a1f"
server_key_version: "encryptionkey@0"
I20260807 02:22:07.613137 9786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "898a3f423ddb4d17b0ba433b35fa1b40"
format_stamp: "Formatted at 2026-08-07 02:22:07 on dist-test-slave-69rg"
server_key: "966d9434058b8584296025c29e91274c"
server_key_iv: "bd455bd57827a4543627c0b720164a1f"
server_key_version: "encryptionkey@0"
I20260807 02:22:07.617048 9786 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 02:22:07.619665 9801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:07.620987 9786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:22:07.621161 9786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "898a3f423ddb4d17b0ba433b35fa1b40"
format_stamp: "Formatted at 2026-08-07 02:22:07 on dist-test-slave-69rg"
server_key: "966d9434058b8584296025c29e91274c"
server_key_iv: "bd455bd57827a4543627c0b720164a1f"
server_key_version: "encryptionkey@0"
I20260807 02:22:07.621294 9786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:07.626065 9786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:07.626729 9786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:07.626935 9786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:07.627506 9786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:07.628455 9786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:07.628531 9786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:07.628600 9786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:07.628662 9786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:07.637949 9786 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:32955
I20260807 02:22:07.637993 9916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:32955 every 8 connection(s)
I20260807 02:22:07.638914 9786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:22:07.641566 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 9786
I20260807 02:22:07.641712 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:22:07.642037 4981 external_mini_cluster.cc:1428] Setting key bc47be1e2fa1afae034a0fe8b4bb0d66
I20260807 02:22:07.644289 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:34959
--builtin_ntp_servers=127.4.221.84:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260807 02:22:07.645881 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.640829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:47725 (local address 127.4.221.126:34959)
0807 02:22:07.640978 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:07.640982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:07.642216 (+ 1234us) server_negotiation.cc:407] Connection header received
0807 02:22:07.643875 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:07.643878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:07.643918 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:07.644003 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:07.645269 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:07.645276 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:07.645282 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:07.645299 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:07.645328 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:07.645342 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:07.645441 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:07.645598 (+ 157us) server_negotiation.cc:299] Negotiation successful
0807 02:22:07.645658 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:22:07.646849 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.641245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:34959 (local address 127.4.221.65:47725)
0807 02:22:07.642051 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:07.642106 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:07.643644 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:07.644169 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:07.644178 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:07.644269 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:07.644954 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:22:07.644956 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:07.645011 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:22:07.645098 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:07.645439 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:07.645447 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:07.645454 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 02:22:07.645629 (+ 175us) client_negotiation.cc:240] Negotiation successful
0807 02:22:07.645970 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":142,"threads_started":1}
I20260807 02:22:07.648324 9917 heartbeater.cc:347] Connected to a master server at 127.4.221.126:34959
I20260807 02:22:07.648600 9917 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:07.649142 9917 heartbeater.cc:511] Master 127.4.221.126:34959 requested a full tablet report, sending...
I20260807 02:22:07.651180 9725 ts_manager.cc:194] Registered new tserver with Master: 898a3f423ddb4d17b0ba433b35fa1b40 (127.4.221.65:32955)
W20260807 02:22:07.767699 9920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:07.768040 9920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:07.768111 9920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:07.772609 9920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:07.772723 9920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:07.772832 9920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:22:07.778604 9920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:34959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:07.779860 9920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:07.781364 9920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:07.792037 9927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:07.792047 9928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:07.792172 9930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:07.792472 9920 server_base.cc:1034] running on GCE node
I20260807 02:22:07.792910 9920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:07.793681 9920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:07.794845 9920 hybrid_clock.cc:648] HybridClock initialized: now 1786069327794827 us; error 32 us; skew 500 ppm
I20260807 02:22:07.796973 9920 webserver.cc:460] Webserver started at http://127.4.221.66:42745/ using document root <none> and password file <none>
I20260807 02:22:07.797537 9920 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:07.797598 9920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:07.797778 9920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:07.799898 9920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "1218833276a941359ec8c508411f0efa"
format_stamp: "Formatted at 2026-08-07 02:22:07 on dist-test-slave-69rg"
server_key: "628ae5ef478a9f9449329477c783420f"
server_key_iv: "aec6c8502d4abda3ad2bf83b11d9f4cd"
server_key_version: "encryptionkey@0"
I20260807 02:22:07.800414 9920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "1218833276a941359ec8c508411f0efa"
format_stamp: "Formatted at 2026-08-07 02:22:07 on dist-test-slave-69rg"
server_key: "628ae5ef478a9f9449329477c783420f"
server_key_iv: "aec6c8502d4abda3ad2bf83b11d9f4cd"
server_key_version: "encryptionkey@0"
I20260807 02:22:07.804060 9920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 02:22:07.806447 9936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:07.807704 9920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:22:07.807824 9920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "1218833276a941359ec8c508411f0efa"
format_stamp: "Formatted at 2026-08-07 02:22:07 on dist-test-slave-69rg"
server_key: "628ae5ef478a9f9449329477c783420f"
server_key_iv: "aec6c8502d4abda3ad2bf83b11d9f4cd"
server_key_version: "encryptionkey@0"
I20260807 02:22:07.807919 9920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:07.812561 9920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:07.813144 9920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:07.813287 9920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:07.813786 9920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:07.814697 9920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:07.814745 9920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:07.814791 9920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:07.814812 9920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:07.823740 9920 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:36091
I20260807 02:22:07.823762 10052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:36091 every 8 connection(s)
I20260807 02:22:07.824790 9920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:22:07.831259 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.826859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:40893 (local address 127.4.221.126:34959)
0807 02:22:07.827012 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:07.827016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:07.828081 (+ 1065us) server_negotiation.cc:407] Connection header received
0807 02:22:07.829451 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:07.829455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:07.829506 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:07.829617 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:07.830792 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:07.830795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:07.830797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:07.830809 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:07.830828 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:07.830836 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:07.830939 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:07.831067 (+ 128us) server_negotiation.cc:299] Negotiation successful
0807 02:22:07.831115 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 02:22:07.832101 10055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:07.827181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:34959 (local address 127.4.221.66:40893)
0807 02:22:07.827904 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:07.827946 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:07.829203 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:07.829788 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:07.829795 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:07.829936 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:07.830525 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:22:07.830527 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:07.830567 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:22:07.830632 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:07.830919 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:07.830928 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:07.830933 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 02:22:07.831112 (+ 179us) client_negotiation.cc:240] Negotiation successful
0807 02:22:07.831356 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":126,"threads_started":1}
I20260807 02:22:07.832793 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 9920
I20260807 02:22:07.832931 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:22:07.833218 4981 external_mini_cluster.cc:1428] Setting key 48a0cfc56da0b5be6318be5deda96825
I20260807 02:22:07.833321 10053 heartbeater.cc:347] Connected to a master server at 127.4.221.126:34959
I20260807 02:22:07.833544 10053 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:07.834048 10053 heartbeater.cc:511] Master 127.4.221.126:34959 requested a full tablet report, sending...
I20260807 02:22:07.835121 9725 ts_manager.cc:194] Registered new tserver with Master: 1218833276a941359ec8c508411f0efa (127.4.221.66:36091)
I20260807 02:22:07.835418 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:34959
--builtin_ntp_servers=127.4.221.84:38509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260807 02:22:07.954656 10056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:07.955010 10056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:07.955086 10056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:07.959439 10056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:07.959532 10056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:07.959625 10056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:22:07.964725 10056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:34959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:07.966081 10056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:07.967549 10056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:07.978617 10065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:07.978638 10062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:07.978617 10061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:07.978842 10056 server_base.cc:1034] running on GCE node
I20260807 02:22:07.979553 10056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:07.980283 10056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:07.981462 10056 hybrid_clock.cc:648] HybridClock initialized: now 1786069327981427 us; error 47 us; skew 500 ppm
I20260807 02:22:07.983654 10056 webserver.cc:460] Webserver started at http://127.4.221.67:37333/ using document root <none> and password file <none>
I20260807 02:22:07.984295 10056 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:07.984369 10056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:07.984603 10056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:07.987206 10056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "84a0adcaec43433e9a8b2e724616b45d"
format_stamp: "Formatted at 2026-08-07 02:22:07 on dist-test-slave-69rg"
server_key: "2fe8e729d91dfff4c8b6fb6737fead29"
server_key_iv: "b9e7a0c530b5c1958e5560a8b1a3472d"
server_key_version: "encryptionkey@0"
I20260807 02:22:07.987983 10056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "84a0adcaec43433e9a8b2e724616b45d"
format_stamp: "Formatted at 2026-08-07 02:22:07 on dist-test-slave-69rg"
server_key: "2fe8e729d91dfff4c8b6fb6737fead29"
server_key_iv: "b9e7a0c530b5c1958e5560a8b1a3472d"
server_key_version: "encryptionkey@0"
I20260807 02:22:07.992318 10056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260807 02:22:07.995558 10072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:07.997009 10056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:22:07.997180 10056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "84a0adcaec43433e9a8b2e724616b45d"
format_stamp: "Formatted at 2026-08-07 02:22:07 on dist-test-slave-69rg"
server_key: "2fe8e729d91dfff4c8b6fb6737fead29"
server_key_iv: "b9e7a0c530b5c1958e5560a8b1a3472d"
server_key_version: "encryptionkey@0"
I20260807 02:22:07.997327 10056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:08.002399 10056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:08.003123 10056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:08.003333 10056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:08.003968 10056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:08.005018 10056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:08.005095 10056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:08.005172 10056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:08.005224 10056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:08.015304 10056 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:32947
I20260807 02:22:08.015333 10189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:32947 every 8 connection(s)
I20260807 02:22:08.016322 10056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:22:08.022997 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.018405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:59831 (local address 127.4.221.126:34959)
0807 02:22:08.018577 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:08.018582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:08.019834 (+ 1252us) server_negotiation.cc:407] Connection header received
0807 02:22:08.021162 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:08.021166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:08.021207 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:08.021290 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:08.022582 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:08.022585 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:08.022587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:08.022599 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:08.022621 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:08.022630 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:08.022718 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:08.022841 (+ 123us) server_negotiation.cc:299] Negotiation successful
0807 02:22:08.022884 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 02:22:08.023366 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 10056
I20260807 02:22:08.023491 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:22:08.023820 4981 external_mini_cluster.cc:1428] Setting key 05c2cd03f337d5dee29cd14d1dd48703
I20260807 02:22:08.024303 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.018914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:34959 (local address 127.4.221.67:59831)
0807 02:22:08.019660 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:08.019703 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:08.020944 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:08.021506 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:08.021513 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:08.021597 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:08.022303 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:22:08.022305 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:08.022345 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:22:08.022414 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:08.022713 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:08.022719 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:08.022723 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 02:22:08.022859 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 02:22:08.023107 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":122,"threads_started":1}
I20260807 02:22:08.025548 10190 heartbeater.cc:347] Connected to a master server at 127.4.221.126:34959
I20260807 02:22:08.025831 10190 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:08.026348 10190 heartbeater.cc:511] Master 127.4.221.126:34959 requested a full tablet report, sending...
I20260807 02:22:08.027472 9725 ts_manager.cc:194] Registered new tserver with Master: 84a0adcaec43433e9a8b2e724616b45d (127.4.221.67:32947)
I20260807 02:22:08.038890 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:22:08.042966 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.041072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.4.221.126:34959)
0807 02:22:08.041232 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:08.041238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:08.041466 (+ 228us) server_negotiation.cc:407] Connection header received
0807 02:22:08.041657 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:08.041659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:08.041700 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:08.041788 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:08.042546 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:08.042552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:08.042560 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:08.042571 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:08.042591 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:08.042599 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:08.042694 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:08.042765 (+ 71us) server_negotiation.cc:299] Negotiation successful
0807 02:22:08.042812 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 02:22:08.046322 9725 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:22:08.049522 9725 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:22:08.064364 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.060035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.4.221.67:32947)
0807 02:22:08.060393 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:08.060412 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:08.060437 (+ 25us) server_negotiation.cc:407] Connection header received
0807 02:22:08.060494 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:08.060500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:08.060862 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:08.061013 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:08.063563 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:08.063571 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:08.063586 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:08.063600 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:08.063649 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:08.063685 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:08.063773 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:08.064038 (+ 265us) server_negotiation.cc:299] Negotiation successful
0807 02:22:08.064158 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":162,"threads_started":1}
I20260807 02:22:08.064755 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.061445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:32955 (local address 127.0.0.1:43400)
0807 02:22:08.061870 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:08.061886 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:08.061967 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:08.063007 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:08.063009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:08.063025 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:08.063436 (+ 411us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:22:08.063440 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:08.063455 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:22:08.063484 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:08.064433 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:08.064437 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:08.064441 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 02:22:08.064507 (+ 66us) client_negotiation.cc:240] Negotiation successful
0807 02:22:08.064587 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":119,"threads_started":1}
I20260807 02:22:08.064756 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.059297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:32947 (local address 127.0.0.1:51394)
0807 02:22:08.059999 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:08.060065 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:08.060183 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:08.061986 (+ 1803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:08.061995 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:08.062080 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:08.062781 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:22:08.062783 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:08.062805 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:22:08.062880 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:08.063854 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:08.063867 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:08.063876 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 02:22:08.064615 (+ 739us) client_negotiation.cc:240] Negotiation successful
0807 02:22:08.064644 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":108,"threads_started":1}
I20260807 02:22:08.065488 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.061622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43400 (local address 127.4.221.65:32955)
0807 02:22:08.062509 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:08.062522 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:08.062554 (+ 32us) server_negotiation.cc:407] Connection header received
0807 02:22:08.062621 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:08.062628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:08.062843 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:08.063261 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:08.064222 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:08.064228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:08.064239 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:08.064254 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:08.064295 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:08.064317 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:08.065195 (+ 878us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:08.065248 (+ 53us) server_negotiation.cc:299] Negotiation successful
0807 02:22:08.065324 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":117,"threads_started":1}
I20260807 02:22:08.068248 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.059480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:36091 (local address 127.0.0.1:55708)
0807 02:22:08.061711 (+ 2231us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:08.061726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:08.061844 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:08.066844 (+ 5000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:08.066848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:08.066867 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:08.067406 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:22:08.067408 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:08.067421 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:22:08.067455 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:08.067758 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:08.067772 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:08.067775 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 02:22:08.068038 (+ 263us) client_negotiation.cc:240] Negotiation successful
0807 02:22:08.068084 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2162,"thread_start_us":117,"threads_started":1}
I20260807 02:22:08.068743 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.060599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55708 (local address 127.4.221.66:36091)
0807 02:22:08.066370 (+ 5771us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:08.066388 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:08.066419 (+ 31us) server_negotiation.cc:407] Connection header received
0807 02:22:08.066476 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:08.066481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:08.066700 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:08.066854 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:08.067593 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:08.067599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:08.067611 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:08.067621 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:08.067658 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:08.067680 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:08.067792 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:08.068486 (+ 694us) server_negotiation.cc:299] Negotiation successful
0807 02:22:08.068569 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5626,"thread_start_us":166,"threads_started":1}
I20260807 02:22:08.068390 9851 tablet_service.cc:1467] Processing CreateTablet for tablet a5bd73a98def41409d65cd238e902667 (DEFAULT_TABLE table=test-table [id=beb083a26ef5401aab79b3b270f265ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:08.068982 10123 tablet_service.cc:1467] Processing CreateTablet for tablet a5bd73a98def41409d65cd238e902667 (DEFAULT_TABLE table=test-table [id=beb083a26ef5401aab79b3b270f265ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:08.069697 9851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5bd73a98def41409d65cd238e902667. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:08.070003 10123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5bd73a98def41409d65cd238e902667. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:08.077260 9986 tablet_service.cc:1467] Processing CreateTablet for tablet a5bd73a98def41409d65cd238e902667 (DEFAULT_TABLE table=test-table [id=beb083a26ef5401aab79b3b270f265ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:08.078223 10207 tablet_bootstrap.cc:492] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d: Bootstrap starting.
I20260807 02:22:08.078538 9986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5bd73a98def41409d65cd238e902667. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:08.080798 10207 tablet_bootstrap.cc:654] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:08.081684 10207 log.cc:824] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:08.084540 10207 tablet_bootstrap.cc:492] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d: No bootstrap required, opened a new log
I20260807 02:22:08.084750 10207 ts_tablet_manager.cc:1397] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260807 02:22:08.085230 10209 tablet_bootstrap.cc:492] T a5bd73a98def41409d65cd238e902667 P 898a3f423ddb4d17b0ba433b35fa1b40: Bootstrap starting.
I20260807 02:22:08.087785 10209 tablet_bootstrap.cc:654] T a5bd73a98def41409d65cd238e902667 P 898a3f423ddb4d17b0ba433b35fa1b40: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:08.087817 10210 tablet_bootstrap.cc:492] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa: Bootstrap starting.
I20260807 02:22:08.088789 10209 log.cc:824] T a5bd73a98def41409d65cd238e902667 P 898a3f423ddb4d17b0ba433b35fa1b40: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:08.089260 10207 raft_consensus.cc:348] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a0adcaec43433e9a8b2e724616b45d" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 32947 } } peers { permanent_uuid: "1218833276a941359ec8c508411f0efa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36091 } } peers { permanent_uuid: "898a3f423ddb4d17b0ba433b35fa1b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 32955 } }
I20260807 02:22:08.089557 10207 raft_consensus.cc:374] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:08.089623 10207 raft_consensus.cc:729] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a0adcaec43433e9a8b2e724616b45d, State: Initialized, Role: FOLLOWER
I20260807 02:22:08.090065 10210 tablet_bootstrap.cc:654] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:08.090741 10207 consensus_queue.cc:260] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a0adcaec43433e9a8b2e724616b45d" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 32947 } } peers { permanent_uuid: "1218833276a941359ec8c508411f0efa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36091 } } peers { permanent_uuid: "898a3f423ddb4d17b0ba433b35fa1b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 32955 } }
I20260807 02:22:08.090983 10210 log.cc:824] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:08.091501 10190 heartbeater.cc:503] Master 127.4.221.126:34959 was elected leader, sending a full tablet report...
I20260807 02:22:08.091945 10207 ts_tablet_manager.cc:1428] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 02:22:08.093106 10210 tablet_bootstrap.cc:492] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa: No bootstrap required, opened a new log
I20260807 02:22:08.093137 10209 tablet_bootstrap.cc:492] T a5bd73a98def41409d65cd238e902667 P 898a3f423ddb4d17b0ba433b35fa1b40: No bootstrap required, opened a new log
I20260807 02:22:08.093336 10210 ts_tablet_manager.cc:1397] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 02:22:08.093338 10209 ts_tablet_manager.cc:1397] T a5bd73a98def41409d65cd238e902667 P 898a3f423ddb4d17b0ba433b35fa1b40: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260807 02:22:08.097926 10209 raft_consensus.cc:348] T a5bd73a98def41409d65cd238e902667 P 898a3f423ddb4d17b0ba433b35fa1b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a0adcaec43433e9a8b2e724616b45d" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 32947 } } peers { permanent_uuid: "1218833276a941359ec8c508411f0efa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36091 } } peers { permanent_uuid: "898a3f423ddb4d17b0ba433b35fa1b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 32955 } }
I20260807 02:22:08.097932 10210 raft_consensus.cc:348] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a0adcaec43433e9a8b2e724616b45d" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 32947 } } peers { permanent_uuid: "1218833276a941359ec8c508411f0efa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36091 } } peers { permanent_uuid: "898a3f423ddb4d17b0ba433b35fa1b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 32955 } }
I20260807 02:22:08.098239 10210 raft_consensus.cc:374] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:08.098237 10209 raft_consensus.cc:374] T a5bd73a98def41409d65cd238e902667 P 898a3f423ddb4d17b0ba433b35fa1b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:08.098294 10210 raft_consensus.cc:729] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1218833276a941359ec8c508411f0efa, State: Initialized, Role: FOLLOWER
I20260807 02:22:08.098297 10209 raft_consensus.cc:729] T a5bd73a98def41409d65cd238e902667 P 898a3f423ddb4d17b0ba433b35fa1b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 898a3f423ddb4d17b0ba433b35fa1b40, State: Initialized, Role: FOLLOWER
I20260807 02:22:08.098781 10209 consensus_queue.cc:260] T a5bd73a98def41409d65cd238e902667 P 898a3f423ddb4d17b0ba433b35fa1b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a0adcaec43433e9a8b2e724616b45d" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 32947 } } peers { permanent_uuid: "1218833276a941359ec8c508411f0efa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36091 } } peers { permanent_uuid: "898a3f423ddb4d17b0ba433b35fa1b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 32955 } }
I20260807 02:22:08.098781 10210 consensus_queue.cc:260] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a0adcaec43433e9a8b2e724616b45d" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 32947 } } peers { permanent_uuid: "1218833276a941359ec8c508411f0efa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36091 } } peers { permanent_uuid: "898a3f423ddb4d17b0ba433b35fa1b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 32955 } }
I20260807 02:22:08.099594 9917 heartbeater.cc:503] Master 127.4.221.126:34959 was elected leader, sending a full tablet report...
I20260807 02:22:08.099845 10053 heartbeater.cc:503] Master 127.4.221.126:34959 was elected leader, sending a full tablet report...
I20260807 02:22:08.099958 10209 ts_tablet_manager.cc:1428] T a5bd73a98def41409d65cd238e902667 P 898a3f423ddb4d17b0ba433b35fa1b40: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 02:22:08.099953 10210 ts_tablet_manager.cc:1428] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260807 02:22:08.140298 9918 tablet.cc:2369] T a5bd73a98def41409d65cd238e902667 P 898a3f423ddb4d17b0ba433b35fa1b40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:22:08.159351 10212 raft_consensus.cc:482] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:22:08.159606 10212 raft_consensus.cc:504] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a0adcaec43433e9a8b2e724616b45d" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 32947 } } peers { permanent_uuid: "1218833276a941359ec8c508411f0efa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36091 } } peers { permanent_uuid: "898a3f423ddb4d17b0ba433b35fa1b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 32955 } }
I20260807 02:22:08.160866 10212 leader_election.cc:288] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1218833276a941359ec8c508411f0efa (127.4.221.66:36091), 898a3f423ddb4d17b0ba433b35fa1b40 (127.4.221.65:32955)
I20260807 02:22:08.163192 10192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.161165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:32955 (local address 127.4.221.67:49815)
0807 02:22:08.161275 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:08.161292 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:08.161415 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:08.161876 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:08.161879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:08.161896 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:08.162552 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:22:08.162553 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:08.162563 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:22:08.162588 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:08.162921 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:08.162923 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:08.162925 (+ 2us) client_negotiation.cc:769] Sending connection context
0807 02:22:08.163006 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 02:22:08.163048 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260807 02:22:08.163201 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.161426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:49815 (local address 127.4.221.65:32955)
0807 02:22:08.161550 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:08.161554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:08.161568 (+ 14us) server_negotiation.cc:407] Connection header received
0807 02:22:08.161616 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:08.161619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:08.161659 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:08.161780 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:08.162763 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:08.162766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:08.162768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:08.162804 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:08.162829 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:08.162840 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:08.162950 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:08.162986 (+ 36us) server_negotiation.cc:299] Negotiation successful
0807 02:22:08.163031 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 02:22:08.163331 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.161327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:49849 (local address 127.4.221.66:36091)
0807 02:22:08.161463 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:08.161468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:08.161839 (+ 371us) server_negotiation.cc:407] Connection header received
0807 02:22:08.161895 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:08.161898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:08.161943 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:08.162046 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:08.162861 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:08.162868 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:08.162874 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:08.162891 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:08.162914 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:08.162924 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:08.163016 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:08.163135 (+ 119us) server_negotiation.cc:299] Negotiation successful
0807 02:22:08.163197 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 02:22:08.163262 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.161165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:36091 (local address 127.4.221.67:49849)
0807 02:22:08.161677 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:08.161693 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:08.161787 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:08.162142 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:08.162146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:08.162166 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:08.162687 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:22:08.162689 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:08.162699 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:22:08.162724 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:08.163029 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:08.163032 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:08.163035 (+ 3us) client_negotiation.cc:769] Sending connection context
0807 02:22:08.163109 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 02:22:08.163136 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":117,"thread_start_us":114,"threads_started":1}
I20260807 02:22:08.164091 9871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5bd73a98def41409d65cd238e902667" candidate_uuid: "84a0adcaec43433e9a8b2e724616b45d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "898a3f423ddb4d17b0ba433b35fa1b40" is_pre_election: true
I20260807 02:22:08.164179 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5bd73a98def41409d65cd238e902667" candidate_uuid: "84a0adcaec43433e9a8b2e724616b45d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1218833276a941359ec8c508411f0efa" is_pre_election: true
I20260807 02:22:08.164459 9871 raft_consensus.cc:2454] T a5bd73a98def41409d65cd238e902667 P 898a3f423ddb4d17b0ba433b35fa1b40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84a0adcaec43433e9a8b2e724616b45d in term 0.
I20260807 02:22:08.164499 10007 raft_consensus.cc:2454] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84a0adcaec43433e9a8b2e724616b45d in term 0.
I20260807 02:22:08.165045 10075 leader_election.cc:302] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1218833276a941359ec8c508411f0efa, 84a0adcaec43433e9a8b2e724616b45d, 898a3f423ddb4d17b0ba433b35fa1b40; no voters:
I20260807 02:22:08.165340 10212 raft_consensus.cc:2781] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:22:08.165442 10212 raft_consensus.cc:482] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:22:08.165503 10212 raft_consensus.cc:3037] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:08.166704 10212 raft_consensus.cc:504] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a0adcaec43433e9a8b2e724616b45d" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 32947 } } peers { permanent_uuid: "1218833276a941359ec8c508411f0efa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36091 } } peers { permanent_uuid: "898a3f423ddb4d17b0ba433b35fa1b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 32955 } }
I20260807 02:22:08.167143 10212 leader_election.cc:288] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [CANDIDATE]: Term 1 election: Requested vote from peers 1218833276a941359ec8c508411f0efa (127.4.221.66:36091), 898a3f423ddb4d17b0ba433b35fa1b40 (127.4.221.65:32955)
I20260807 02:22:08.167557 9871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5bd73a98def41409d65cd238e902667" candidate_uuid: "84a0adcaec43433e9a8b2e724616b45d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "898a3f423ddb4d17b0ba433b35fa1b40"
I20260807 02:22:08.167557 10007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5bd73a98def41409d65cd238e902667" candidate_uuid: "84a0adcaec43433e9a8b2e724616b45d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1218833276a941359ec8c508411f0efa"
I20260807 02:22:08.167712 10007 raft_consensus.cc:3037] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:08.167682 9871 raft_consensus.cc:3037] T a5bd73a98def41409d65cd238e902667 P 898a3f423ddb4d17b0ba433b35fa1b40 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:08.168860 9871 raft_consensus.cc:2454] T a5bd73a98def41409d65cd238e902667 P 898a3f423ddb4d17b0ba433b35fa1b40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84a0adcaec43433e9a8b2e724616b45d in term 1.
I20260807 02:22:08.168869 10007 raft_consensus.cc:2454] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84a0adcaec43433e9a8b2e724616b45d in term 1.
I20260807 02:22:08.169250 10073 leader_election.cc:302] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84a0adcaec43433e9a8b2e724616b45d, 898a3f423ddb4d17b0ba433b35fa1b40; no voters:
I20260807 02:22:08.169445 10212 raft_consensus.cc:2781] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:08.169688 10212 raft_consensus.cc:686] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [term 1 LEADER]: Becoming Leader. State: Replica: 84a0adcaec43433e9a8b2e724616b45d, State: Running, Role: LEADER
I20260807 02:22:08.170115 10212 consensus_queue.cc:237] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a0adcaec43433e9a8b2e724616b45d" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 32947 } } peers { permanent_uuid: "1218833276a941359ec8c508411f0efa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36091 } } peers { permanent_uuid: "898a3f423ddb4d17b0ba433b35fa1b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 32955 } }
I20260807 02:22:08.173458 9725 catalog_manager.cc:5495] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d reported cstate change: term changed from 0 to 1, leader changed from <none> to 84a0adcaec43433e9a8b2e724616b45d (127.4.221.67). New cstate: current_term: 1 leader_uuid: "84a0adcaec43433e9a8b2e724616b45d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a0adcaec43433e9a8b2e724616b45d" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 32947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1218833276a941359ec8c508411f0efa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "898a3f423ddb4d17b0ba433b35fa1b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 32955 } health_report { overall_health: UNKNOWN } } }
I20260807 02:22:08.190402 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.188823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.4.221.67:32947)
0807 02:22:08.188956 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:08.188959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:08.188973 (+ 14us) server_negotiation.cc:407] Connection header received
0807 02:22:08.189100 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:08.189102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:08.189165 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:08.189280 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:08.190052 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:08.190056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:08.190058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:08.190070 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:08.190091 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:08.190099 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:08.190188 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:08.190235 (+ 47us) server_negotiation.cc:299] Negotiation successful
0807 02:22:08.190272 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:22:08.197538 10007 raft_consensus.cc:1261] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa [term 1 FOLLOWER]: Refusing update from remote peer 84a0adcaec43433e9a8b2e724616b45d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:08.197561 9871 raft_consensus.cc:1261] T a5bd73a98def41409d65cd238e902667 P 898a3f423ddb4d17b0ba433b35fa1b40 [term 1 FOLLOWER]: Refusing update from remote peer 84a0adcaec43433e9a8b2e724616b45d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:08.198508 10218 consensus_queue.cc:1035] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [LEADER]: Connected to new peer: Peer: permanent_uuid: "898a3f423ddb4d17b0ba433b35fa1b40" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 32955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:08.198745 10212 consensus_queue.cc:1035] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1218833276a941359ec8c508411f0efa" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:08.206557 10223 mvcc.cc:205] Tried to move back new op lower bound from 7315739968287920128 to 7315739968187572224. Current Snapshot: MvccSnapshot[applied={T|T < 7315739968287920128}]
I20260807 02:22:08.210821 10221 mvcc.cc:205] Tried to move back new op lower bound from 7315739968287920128 to 7315739968187572224. Current Snapshot: MvccSnapshot[applied={T|T < 7315739968287920128 or (T in {7315739968287920128})}]
I20260807 02:22:08.213969 10222 mvcc.cc:205] Tried to move back new op lower bound from 7315739968287920128 to 7315739968187572224. Current Snapshot: MvccSnapshot[applied={T|T < 7315739968287920128 or (T in {7315739968287920128})}]
I20260807 02:22:08.215366 9724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:08.215575 9724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:08.218446 9724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e095761372f45a98d6f670292031b64: Sending DeleteTablet for 3 replicas of tablet a5bd73a98def41409d65cd238e902667
I20260807 02:22:08.219420 10123 tablet_service.cc:1514] Processing DeleteTablet for tablet a5bd73a98def41409d65cd238e902667 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:08 UTC) from {username='slave'} at 127.0.0.1:51394
I20260807 02:22:08.219420 9986 tablet_service.cc:1514] Processing DeleteTablet for tablet a5bd73a98def41409d65cd238e902667 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:08 UTC) from {username='slave'} at 127.0.0.1:55708
I20260807 02:22:08.220041 10231 tablet_replica.cc:330] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa: stopping tablet replica
I20260807 02:22:08.220343 10230 tablet_replica.cc:330] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d: stopping tablet replica
I20260807 02:22:08.220371 9851 tablet_service.cc:1514] Processing DeleteTablet for tablet a5bd73a98def41409d65cd238e902667 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:08 UTC) from {username='slave'} at 127.0.0.1:43400
I20260807 02:22:08.220568 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 9786
I20260807 02:22:08.220912 10230 raft_consensus.cc:2229] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:22:08.221601 10231 raft_consensus.cc:2229] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:08.222105 10231 raft_consensus.cc:2258] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:08.222560 10230 raft_consensus.cc:2258] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:08.224448 10230 ts_tablet_manager.cc:1905] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:08.224598 10231 ts_tablet_manager.cc:1905] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:08.227790 10231 ts_tablet_manager.cc:1918] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:08.227923 10231 log.cc:1196] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/a5bd73a98def41409d65cd238e902667
I20260807 02:22:08.228295 10231 ts_tablet_manager.cc:1939] T a5bd73a98def41409d65cd238e902667 P 1218833276a941359ec8c508411f0efa: Deleting consensus metadata
I20260807 02:22:08.229166 10230 ts_tablet_manager.cc:1918] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:08.229391 10230 log.cc:1196] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/a5bd73a98def41409d65cd238e902667
I20260807 02:22:08.229885 10230 ts_tablet_manager.cc:1939] T a5bd73a98def41409d65cd238e902667 P 84a0adcaec43433e9a8b2e724616b45d: Deleting consensus metadata
I20260807 02:22:08.229969 9710 catalog_manager.cc:4841] TS 1218833276a941359ec8c508411f0efa (127.4.221.66:36091): tablet a5bd73a98def41409d65cd238e902667 (table test-table [id=beb083a26ef5401aab79b3b270f265ce]) successfully deleted
I20260807 02:22:08.231329 9711 catalog_manager.cc:4841] TS 84a0adcaec43433e9a8b2e724616b45d (127.4.221.67:32947): tablet a5bd73a98def41409d65cd238e902667 (table test-table [id=beb083a26ef5401aab79b3b270f265ce]) successfully deleted
W20260807 02:22:08.231812 9708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:32955 (error 108)
I20260807 02:22:08.231886 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 9920
I20260807 02:22:08.233873 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.233498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:32955 (local address 127.0.0.1:43408)
0807 02:22:08.233647 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:08.233735 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:32955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260807 02:22:08.234145 9708 catalog_manager.cc:4584] TS 898a3f423ddb4d17b0ba433b35fa1b40 (127.4.221.65:32955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a5bd73a98def41409d65cd238e902667: Network error: Client connection negotiation failed: client connection to 127.4.221.65:32955: connect: Connection refused (error 111)
I20260807 02:22:08.241925 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 10056
I20260807 02:22:08.249758 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 9691
2026-08-07T02:22:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1086 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260807 02:22:08.273533 4981 mini_oidc.cc:152] Starting JWKS server
I20260807 02:22:08.275556 4981 webserver.cc:460] Webserver started at https://127.0.0.1:45421/ using document root <none> and password file <none>
I20260807 02:22:08.275763 4981 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 02:22:08.276082 4981 webserver.cc:460] Webserver started at http://127.0.0.1:40983/ using document root <none> and password file <none>
2026-08-07T02:22:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:08Z Disabled control of system clock
I20260807 02:22:08.287804 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:32863
--webserver_interface=127.4.221.126
--webserver_port=0
--builtin_ntp_servers=127.4.221.84:38767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:32863
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45421/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/testchainca.cert with env {}
W20260807 02:22:08.408507 10238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:08.408838 10238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:08.408883 10238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:08.413110 10238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:08.413210 10238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:08.413228 10238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:08.413245 10238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:08.413261 10238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 02:22:08.413278 10238 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 02:22:08.413292 10238 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45421/jwks.json
I20260807 02:22:08.419364 10238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:32863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:32863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45421/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:08.420527 10238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:08.421540 10238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:08.430657 10246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:08.430841 10238 server_base.cc:1034] running on GCE node
W20260807 02:22:08.430686 10248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:08.430686 10245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:08.431569 10238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:08.432823 10238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:08.434016 10238 hybrid_clock.cc:648] HybridClock initialized: now 1786069328433999 us; error 32 us; skew 500 ppm
I20260807 02:22:08.436036 10238 webserver.cc:460] Webserver started at http://127.4.221.126:38245/ using document root <none> and password file <none>
I20260807 02:22:08.436640 10238 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:08.436748 10238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:08.436991 10238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:08.439206 10238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "dc531a4e7a654a8c8a0217b5e8752593"
format_stamp: "Formatted at 2026-08-07 02:22:08 on dist-test-slave-69rg"
server_key: "26fc06a1bf8d33a79aa4f82d682bd064"
server_key_iv: "fd0150707da54ceca30f2305f26971d5"
server_key_version: "encryptionkey@0"
I20260807 02:22:08.439790 10238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "dc531a4e7a654a8c8a0217b5e8752593"
format_stamp: "Formatted at 2026-08-07 02:22:08 on dist-test-slave-69rg"
server_key: "26fc06a1bf8d33a79aa4f82d682bd064"
server_key_iv: "fd0150707da54ceca30f2305f26971d5"
server_key_version: "encryptionkey@0"
I20260807 02:22:08.443804 10238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260807 02:22:08.446377 10254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:08.447665 10238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:22:08.447821 10238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "dc531a4e7a654a8c8a0217b5e8752593"
format_stamp: "Formatted at 2026-08-07 02:22:08 on dist-test-slave-69rg"
server_key: "26fc06a1bf8d33a79aa4f82d682bd064"
server_key_iv: "fd0150707da54ceca30f2305f26971d5"
server_key_version: "encryptionkey@0"
I20260807 02:22:08.447942 10238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:08.464202 10238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:08.465045 10238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:08.465291 10238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:08.472783 10238 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:32863
I20260807 02:22:08.472797 10307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:32863 every 8 connection(s)
I20260807 02:22:08.473920 10238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:08.475606 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 10238
I20260807 02:22:08.475750 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:22:08.476063 4981 external_mini_cluster.cc:1428] Setting key 0cd62c8b95a7198db08ed2074201fa4e
I20260807 02:22:08.478080 10308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:08.484683 10308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593: Bootstrap starting.
I20260807 02:22:08.485651 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.477508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.4.221.126:32863)
0807 02:22:08.478194 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:08.478217 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:08.478263 (+ 46us) server_negotiation.cc:407] Connection header received
0807 02:22:08.478917 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:08.478932 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:08.479306 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:08.479583 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:08.479937 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:08.480905 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:08.482142 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:08.482653 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:08.482777 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:08.483773 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:08.483778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:08.483789 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:08.484184 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:08.484246 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:08.484265 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:08.484345 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:08.484578 (+ 233us) server_negotiation.cc:299] Negotiation successful
0807 02:22:08.484915 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":140,"threads_started":1}
I20260807 02:22:08.487253 10308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:08.488198 10308 log.cc:824] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:08.488822 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:22:08.490703 10308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593: No bootstrap required, opened a new log
I20260807 02:22:08.494545 10308 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc531a4e7a654a8c8a0217b5e8752593" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32863 } }
I20260807 02:22:08.494807 10308 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:08.494884 10308 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc531a4e7a654a8c8a0217b5e8752593, State: Initialized, Role: FOLLOWER
I20260807 02:22:08.495337 10308 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc531a4e7a654a8c8a0217b5e8752593" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32863 } }
I20260807 02:22:08.495499 10308 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:08.495591 10308 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:08.495697 10308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:08.496630 10308 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc531a4e7a654a8c8a0217b5e8752593" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32863 } }
I20260807 02:22:08.496976 10308 leader_election.cc:302] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc531a4e7a654a8c8a0217b5e8752593; no voters:
I20260807 02:22:08.497370 10308 leader_election.cc:288] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:22:08.497516 10313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:08.497774 10313 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [term 1 LEADER]: Becoming Leader. State: Replica: dc531a4e7a654a8c8a0217b5e8752593, State: Running, Role: LEADER
I20260807 02:22:08.498088 10313 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc531a4e7a654a8c8a0217b5e8752593" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32863 } }
I20260807 02:22:08.498663 10308 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:08.499346 10313 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc531a4e7a654a8c8a0217b5e8752593. Latest consensus state: current_term: 1 leader_uuid: "dc531a4e7a654a8c8a0217b5e8752593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc531a4e7a654a8c8a0217b5e8752593" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32863 } } }
I20260807 02:22:08.499475 10313 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:08.499784 10314 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc531a4e7a654a8c8a0217b5e8752593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc531a4e7a654a8c8a0217b5e8752593" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32863 } } }
I20260807 02:22:08.499895 10314 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:08.499877 10321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:08.502940 10321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:08.508090 10321 catalog_manager.cc:1347] Generated new cluster ID: fb4cc027ad284ed59d553dadb8e70e65
I20260807 02:22:08.508172 10321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:08.534281 10321 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:22:08.535318 10321 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:08.542717 10321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc531a4e7a654a8c8a0217b5e8752593: Generated new TSK 0
I20260807 02:22:08.543663 10321 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 02:22:08.583920 10341 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260807 02:22:08.584096 10341 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 02:22:08.584796 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.566553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.4.221.126:32863)
0807 02:22:08.566749 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:08.566756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:08.567048 (+ 292us) server_negotiation.cc:407] Connection header received
0807 02:22:08.567328 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:08.567331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:08.567389 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:08.567486 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0807 02:22:08.567820 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:08.568368 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:08.569680 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:08.569978 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:08.570062 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:08.570480 (+ 418us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 02:22:08.584400 (+ 13920us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 02:22:08.584669 (+ 269us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55470: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":106}
W20260807 02:22:08.584890 10311 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55470: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260807 02:22:08.586648 10339 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:08.566388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32863 (local address 127.0.0.1:55470)
0807 02:22:08.566940 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:08.566962 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:08.567113 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:08.567535 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:08.567538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:08.567548 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0807 02:22:08.567675 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:08.567682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:08.568518 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:08.568522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:08.569553 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:08.569564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:08.570099 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:08.570103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:08.570200 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:08.570263 (+ 63us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 02:22:08.584690 (+ 14427us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0807 02:22:08.584993 (+ 303us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.221.126:32863: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":140,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.221.126:32863: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45421/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260807 02:22:08.587579 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 10238
2026-08-07T02:22:08Z chronyd exiting
I20260807 02:22:08.627403 4981 test_util.cc:175] -----------------------------------------------
I20260807 02:22:08.627498 4981 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786069301823597-4981-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (356 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260807 02:22:08.628656 4981 mini_oidc.cc:152] Starting JWKS server
I20260807 02:22:08.629528 4981 webserver.cc:460] Webserver started at https://127.0.0.1:39431/ using document root <none> and password file <none>
I20260807 02:22:08.629612 4981 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 02:22:08.629900 4981 webserver.cc:460] Webserver started at http://127.0.0.1:43029/ using document root <none> and password file <none>
2026-08-07T02:22:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:08Z Disabled control of system clock
I20260807 02:22:08.640743 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:40373
--webserver_interface=127.4.221.126
--webserver_port=0
--builtin_ntp_servers=127.4.221.84:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:40373
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39431/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/testchainca.cert with env {}
W20260807 02:22:08.790601 10347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:08.791165 10347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:08.791235 10347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:08.797225 10347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:08.797392 10347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:08.797415 10347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:08.797437 10347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:08.797469 10347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 02:22:08.797492 10347 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 02:22:08.797509 10347 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39431/jwks.json
I20260807 02:22:08.804906 10347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:40373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:40373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39431/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:08.806915 10347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:08.808882 10347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:08.823642 10353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:08.823622 10355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:08.823622 10352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:08.824630 10347 server_base.cc:1034] running on GCE node
I20260807 02:22:08.825464 10347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:08.827165 10347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:08.828357 10347 hybrid_clock.cc:648] HybridClock initialized: now 1786069328828340 us; error 49 us; skew 500 ppm
I20260807 02:22:08.831029 10347 webserver.cc:460] Webserver started at http://127.4.221.126:42573/ using document root <none> and password file <none>
I20260807 02:22:08.831751 10347 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:08.831822 10347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:08.832043 10347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:08.834681 10347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "ba74e98255d94379991c3f1a3b0ce5ba"
format_stamp: "Formatted at 2026-08-07 02:22:08 on dist-test-slave-69rg"
server_key: "a069f3797cd4952b14383ec3a6b28607"
server_key_iv: "503605b590230fcf57db24f158d5c977"
server_key_version: "encryptionkey@0"
I20260807 02:22:08.835309 10347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "ba74e98255d94379991c3f1a3b0ce5ba"
format_stamp: "Formatted at 2026-08-07 02:22:08 on dist-test-slave-69rg"
server_key: "a069f3797cd4952b14383ec3a6b28607"
server_key_iv: "503605b590230fcf57db24f158d5c977"
server_key_version: "encryptionkey@0"
I20260807 02:22:08.840220 10347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:22:08.843835 10361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:08.845700 10347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:22:08.845964 10347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "ba74e98255d94379991c3f1a3b0ce5ba"
format_stamp: "Formatted at 2026-08-07 02:22:08 on dist-test-slave-69rg"
server_key: "a069f3797cd4952b14383ec3a6b28607"
server_key_iv: "503605b590230fcf57db24f158d5c977"
server_key_version: "encryptionkey@0"
I20260807 02:22:08.846130 10347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:08.860284 10347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:08.861354 10347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:08.861625 10347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:08.871955 10347 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:40373
I20260807 02:22:08.872018 10417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:40373 every 8 connection(s)
I20260807 02:22:08.874226 10347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:08.879477 10418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:08.880576 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 10347
I20260807 02:22:08.880795 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:22:08.881178 4981 external_mini_cluster.cc:1428] Setting key 8a43d95356febf013e1214e98c98ac2d
I20260807 02:22:08.888748 10418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba: Bootstrap starting.
I20260807 02:22:08.892386 10418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:08.893607 10418 log.cc:824] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:08.894243 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.883034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.4.221.126:40373)
0807 02:22:08.883963 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:08.883987 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:08.884045 (+ 58us) server_negotiation.cc:407] Connection header received
0807 02:22:08.885027 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:08.885060 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:08.885689 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:08.886182 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:08.886756 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:08.888366 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:08.889473 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:08.890261 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:08.890485 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:08.891517 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:08.891524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:08.891543 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:08.892141 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:08.892254 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:08.892286 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:08.892413 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:08.892579 (+ 166us) server_negotiation.cc:299] Negotiation successful
0807 02:22:08.892950 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":153,"threads_started":1}
I20260807 02:22:08.896728 10418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba: No bootstrap required, opened a new log
I20260807 02:22:08.898196 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:22:08.902032 10418 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba74e98255d94379991c3f1a3b0ce5ba" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40373 } }
I20260807 02:22:08.902457 10418 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:08.902532 10418 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba74e98255d94379991c3f1a3b0ce5ba, State: Initialized, Role: FOLLOWER
I20260807 02:22:08.903143 10418 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba74e98255d94379991c3f1a3b0ce5ba" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40373 } }
I20260807 02:22:08.903326 10418 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:08.903407 10418 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:08.903561 10418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:08.904866 10418 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba74e98255d94379991c3f1a3b0ce5ba" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40373 } }
I20260807 02:22:08.905288 10418 leader_election.cc:302] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba74e98255d94379991c3f1a3b0ce5ba; no voters:
I20260807 02:22:08.905709 10418 leader_election.cc:288] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:22:08.906047 10423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:08.906281 10423 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [term 1 LEADER]: Becoming Leader. State: Replica: ba74e98255d94379991c3f1a3b0ce5ba, State: Running, Role: LEADER
I20260807 02:22:08.906693 10423 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba74e98255d94379991c3f1a3b0ce5ba" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40373 } }
I20260807 02:22:08.907596 10418 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:08.908497 10425 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba74e98255d94379991c3f1a3b0ce5ba. Latest consensus state: current_term: 1 leader_uuid: "ba74e98255d94379991c3f1a3b0ce5ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba74e98255d94379991c3f1a3b0ce5ba" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40373 } } }
I20260807 02:22:08.908749 10425 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:08.908972 10424 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba74e98255d94379991c3f1a3b0ce5ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba74e98255d94379991c3f1a3b0ce5ba" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40373 } } }
I20260807 02:22:08.909088 10424 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:08.909596 10432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:08.913796 10432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:08.921609 10432 catalog_manager.cc:1347] Generated new cluster ID: 04f96608792141d5bd70b8671619d0b8
I20260807 02:22:08.921730 10432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:08.929991 10432 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:22:08.931303 10432 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:08.943230 10432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba74e98255d94379991c3f1a3b0ce5ba: Generated new TSK 0
I20260807 02:22:08.944348 10432 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260807 02:22:08.982757 10449 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260807 02:22:08.982905 10449 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260807 02:22:08.983922 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:08.964383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.4.221.126:40373)
0807 02:22:08.964567 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:08.964573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:08.964829 (+ 256us) server_negotiation.cc:407] Connection header received
0807 02:22:08.965055 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:08.965060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:08.965141 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:08.965292 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0807 02:22:08.965651 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:08.966636 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:08.967762 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:08.968143 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:08.968258 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:08.968674 (+ 416us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 02:22:08.983318 (+ 14644us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 02:22:08.983712 (+ 394us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35298: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260807 02:22:08.984005 10421 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35298: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260807 02:22:08.984058 10448 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:08.964213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40373 (local address 127.0.0.1:35298)
0807 02:22:08.964713 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:08.964729 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:08.964873 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:08.965328 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:08.965332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:08.965342 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0807 02:22:08.965511 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:08.965517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:08.966798 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:08.966803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:08.967620 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:08.967634 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:08.968287 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:08.968291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:08.968388 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:08.968422 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 02:22:08.983634 (+ 15212us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0807 02:22:08.983845 (+ 211us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.221.126:40373: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":163,"threads_started":1}
I20260807 02:22:08.985047 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 10347
2026-08-07T02:22:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (400 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260807 02:22:09.029234 4981 mini_oidc.cc:152] Starting JWKS server
I20260807 02:22:09.030449 4981 webserver.cc:460] Webserver started at https://127.0.0.1:33193/ using document root <none> and password file <none>
I20260807 02:22:09.030612 4981 mini_oidc.cc:189] Starting OIDC Discovery server
I20260807 02:22:09.030920 4981 webserver.cc:460] Webserver started at http://127.0.0.1:42111/ using document root <none> and password file <none>
2026-08-07T02:22:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:09Z Disabled control of system clock
I20260807 02:22:09.046917 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44353
--webserver_interface=127.4.221.126
--webserver_port=0
--builtin_ntp_servers=127.4.221.84:40813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:44353
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33193/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 02:22:09.191648 10455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:09.192085 10455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:09.192135 10455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:09.196719 10455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:09.196825 10455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:09.196846 10455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:09.196874 10455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:09.196894 10455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 02:22:09.196916 10455 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 02:22:09.196933 10455 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33193/jwks.json
I20260807 02:22:09.202538 10455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:40813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:44353
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33193/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:09.204177 10455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:09.206132 10455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:09.217700 10461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:09.217706 10460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:09.217736 10463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:09.218025 10455 server_base.cc:1034] running on GCE node
I20260807 02:22:09.219090 10455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:09.220801 10455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:09.222018 10455 hybrid_clock.cc:648] HybridClock initialized: now 1786069329221990 us; error 48 us; skew 500 ppm
I20260807 02:22:09.224763 10455 webserver.cc:460] Webserver started at http://127.4.221.126:41955/ using document root <none> and password file <none>
I20260807 02:22:09.225607 10455 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:09.225725 10455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:09.226109 10455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:09.229275 10455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "e9bf548032e645b9837b0b6b1e7a53be"
format_stamp: "Formatted at 2026-08-07 02:22:09 on dist-test-slave-69rg"
server_key: "860239e02d68eb317bf798b8e569a5d2"
server_key_iv: "56e5b18f99ad97bdcfe7c8535c85ce6c"
server_key_version: "encryptionkey@0"
I20260807 02:22:09.230149 10455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "e9bf548032e645b9837b0b6b1e7a53be"
format_stamp: "Formatted at 2026-08-07 02:22:09 on dist-test-slave-69rg"
server_key: "860239e02d68eb317bf798b8e569a5d2"
server_key_iv: "56e5b18f99ad97bdcfe7c8535c85ce6c"
server_key_version: "encryptionkey@0"
I20260807 02:22:09.235450 10455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 02:22:09.239183 10469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:09.240919 10455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:22:09.241132 10455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "e9bf548032e645b9837b0b6b1e7a53be"
format_stamp: "Formatted at 2026-08-07 02:22:09 on dist-test-slave-69rg"
server_key: "860239e02d68eb317bf798b8e569a5d2"
server_key_iv: "56e5b18f99ad97bdcfe7c8535c85ce6c"
server_key_version: "encryptionkey@0"
I20260807 02:22:09.241297 10455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:09.255146 10455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:09.256099 10455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:09.256341 10455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:09.265789 10455 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:44353
I20260807 02:22:09.265887 10523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:44353 every 8 connection(s)
I20260807 02:22:09.267275 10455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:09.267768 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 10455
I20260807 02:22:09.267951 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:22:09.268409 4981 external_mini_cluster.cc:1428] Setting key ac2813ca0742c11b51ddb292cf438ff8
I20260807 02:22:09.272717 10524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:09.281507 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:09.270542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.4.221.126:44353)
0807 02:22:09.271464 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:09.271493 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:09.271546 (+ 53us) server_negotiation.cc:407] Connection header received
0807 02:22:09.272453 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:09.272487 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:09.273119 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:09.273563 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:09.274128 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:09.275489 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:09.277032 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:09.277674 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:09.277912 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:09.279072 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:09.279078 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:09.279098 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:09.279628 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:09.279707 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:09.279732 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:09.279831 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:09.280081 (+ 250us) server_negotiation.cc:299] Negotiation successful
0807 02:22:09.280492 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":205,"threads_started":1}
I20260807 02:22:09.282102 10524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be: Bootstrap starting.
I20260807 02:22:09.285254 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:22:09.285487 10524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:09.286521 10524 log.cc:824] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:09.289398 10524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be: No bootstrap required, opened a new log
I20260807 02:22:09.294145 10524 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bf548032e645b9837b0b6b1e7a53be" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44353 } }
I20260807 02:22:09.294555 10524 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:09.294615 10524 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9bf548032e645b9837b0b6b1e7a53be, State: Initialized, Role: FOLLOWER
I20260807 02:22:09.295182 10524 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bf548032e645b9837b0b6b1e7a53be" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44353 } }
I20260807 02:22:09.295348 10524 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:09.295418 10524 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:09.295521 10524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:09.309719 10524 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bf548032e645b9837b0b6b1e7a53be" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44353 } }
I20260807 02:22:09.310649 10524 leader_election.cc:302] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9bf548032e645b9837b0b6b1e7a53be; no voters:
I20260807 02:22:09.311295 10524 leader_election.cc:288] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:22:09.311491 10529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:09.311817 10529 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [term 1 LEADER]: Becoming Leader. State: Replica: e9bf548032e645b9837b0b6b1e7a53be, State: Running, Role: LEADER
I20260807 02:22:09.312328 10529 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bf548032e645b9837b0b6b1e7a53be" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44353 } }
I20260807 02:22:09.313763 10524 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:09.314563 10531 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9bf548032e645b9837b0b6b1e7a53be. Latest consensus state: current_term: 1 leader_uuid: "e9bf548032e645b9837b0b6b1e7a53be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bf548032e645b9837b0b6b1e7a53be" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44353 } } }
I20260807 02:22:09.314740 10531 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:09.315173 10532 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9bf548032e645b9837b0b6b1e7a53be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9bf548032e645b9837b0b6b1e7a53be" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44353 } } }
I20260807 02:22:09.315307 10532 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:09.315697 10534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:09.320152 10534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:09.328526 10534 catalog_manager.cc:1347] Generated new cluster ID: 5bbc72f5350d4bd3b8229ef4d7171eef
I20260807 02:22:09.328675 10534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:09.347663 10534 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:22:09.348908 10534 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:09.370498 10534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9bf548032e645b9837b0b6b1e7a53be: Generated new TSK 0
I20260807 02:22:09.371702 10534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:22:09.402958 10527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:09.395976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.4.221.126:44353)
0807 02:22:09.396167 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:09.396174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:09.396435 (+ 261us) server_negotiation.cc:407] Connection header received
0807 02:22:09.396676 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:09.396680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:09.396767 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:09.396928 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0807 02:22:09.397261 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:09.398203 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:09.399261 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:09.399652 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:09.399779 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:09.400236 (+ 457us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 02:22:09.402391 (+ 2155us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33193/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 02:22:09.402736 (+ 345us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33193/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":72}
W20260807 02:22:09.403085 10527 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37084: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33193/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260807 02:22:09.404513 10555 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:09.395804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44353 (local address 127.0.0.1:37084)
0807 02:22:09.396314 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:09.396330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:09.396483 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:09.396953 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:09.396956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:09.396965 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0807 02:22:09.397112 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:09.397118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:09.398391 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:09.398394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:09.399130 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:09.399141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:09.399836 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:09.399839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:09.399926 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:09.399958 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 02:22:09.402631 (+ 2673us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33193/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0807 02:22:09.404276 (+ 1645us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.221.126:44353: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33193/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":1357,"thread_start_us":160,"threads_started":1}
I20260807 02:22:09.405445 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 10455
E20260807 02:22:09.418787 10556 webserver.cc:533] Webserver: sslize failed:
E20260807 02:22:09.418921 10556 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-07T02:22:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (423 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-07T02:22:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:09Z Disabled control of system clock
I20260807 02:22:09.465314 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:38239
--webserver_interface=127.4.221.126
--webserver_port=0
--builtin_ntp_servers=127.4.221.84:36387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:38239
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/testchainca.cert with env {}
W20260807 02:22:09.618871 10560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:09.619309 10560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:09.619369 10560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:09.624403 10560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:09.624500 10560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:09.624519 10560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:09.624541 10560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:09.624562 10560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260807 02:22:09.624619 10560 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260807 02:22:09.624657 10560 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260807 02:22:09.631210 10560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:38239
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:38239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:09.632822 10560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:09.634306 10560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:09.646739 10568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:09.646970 10560 server_base.cc:1034] running on GCE node
W20260807 02:22:09.646761 10565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:09.646776 10566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:09.647992 10560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:09.649662 10560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:09.650906 10560 hybrid_clock.cc:648] HybridClock initialized: now 1786069329650886 us; error 42 us; skew 500 ppm
I20260807 02:22:09.653615 10560 webserver.cc:460] Webserver started at http://127.4.221.126:39735/ using document root <none> and password file <none>
I20260807 02:22:09.654505 10560 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:09.654624 10560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:09.654940 10560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:09.657794 10560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "6f33265312a042c79002151963f04605"
format_stamp: "Formatted at 2026-08-07 02:22:09 on dist-test-slave-69rg"
server_key: "e6cd9db3ee1da0ae73ae039baaa12def"
server_key_iv: "c61de204f32bf4c1a2c2c53d0c588d26"
server_key_version: "encryptionkey@0"
I20260807 02:22:09.658710 10560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "6f33265312a042c79002151963f04605"
format_stamp: "Formatted at 2026-08-07 02:22:09 on dist-test-slave-69rg"
server_key: "e6cd9db3ee1da0ae73ae039baaa12def"
server_key_iv: "c61de204f32bf4c1a2c2c53d0c588d26"
server_key_version: "encryptionkey@0"
I20260807 02:22:09.664227 10560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:22:09.668927 10575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:09.671054 10560 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260807 02:22:09.671286 10560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "6f33265312a042c79002151963f04605"
format_stamp: "Formatted at 2026-08-07 02:22:09 on dist-test-slave-69rg"
server_key: "e6cd9db3ee1da0ae73ae039baaa12def"
server_key_iv: "c61de204f32bf4c1a2c2c53d0c588d26"
server_key_version: "encryptionkey@0"
I20260807 02:22:09.671504 10560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:09.691095 10560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:09.692130 10560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:09.692394 10560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:09.701596 10560 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:38239
I20260807 02:22:09.701622 10629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:38239 every 8 connection(s)
I20260807 02:22:09.703191 10560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:09.704952 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 10560
I20260807 02:22:09.705117 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:22:09.705466 4981 external_mini_cluster.cc:1428] Setting key cce7b799c4378a84598429b1808b07c5
I20260807 02:22:09.708469 10630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:09.716995 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:09.707241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.4.221.126:38239)
0807 02:22:09.708037 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:09.708056 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:09.708102 (+ 46us) server_negotiation.cc:407] Connection header received
0807 02:22:09.708932 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:09.708952 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:09.709415 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:09.709826 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:09.710393 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:09.711913 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:09.712911 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:09.713481 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:09.713699 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:09.714812 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:09.714817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:09.714834 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:09.715314 (+ 480us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:09.715385 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:09.715409 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:09.715544 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:09.715684 (+ 140us) server_negotiation.cc:299] Negotiation successful
0807 02:22:09.716037 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":181,"threads_started":1}
I20260807 02:22:09.718152 10630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605: Bootstrap starting.
I20260807 02:22:09.720728 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:22:09.721351 10630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:09.722424 10630 log.cc:824] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:09.724846 10630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605: No bootstrap required, opened a new log
I20260807 02:22:09.729357 10630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f33265312a042c79002151963f04605" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38239 } }
I20260807 02:22:09.729647 10630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:09.729723 10630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f33265312a042c79002151963f04605, State: Initialized, Role: FOLLOWER
I20260807 02:22:09.730304 10630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f33265312a042c79002151963f04605" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38239 } }
I20260807 02:22:09.730485 10630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:09.730592 10630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:09.730684 10630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:09.809466 10630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f33265312a042c79002151963f04605" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38239 } }
I20260807 02:22:09.810190 10630 leader_election.cc:302] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f33265312a042c79002151963f04605; no voters:
I20260807 02:22:09.810725 10630 leader_election.cc:288] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:22:09.810961 10635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:09.811260 10635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [term 1 LEADER]: Becoming Leader. State: Replica: 6f33265312a042c79002151963f04605, State: Running, Role: LEADER
I20260807 02:22:09.812150 10635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f33265312a042c79002151963f04605" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38239 } }
I20260807 02:22:09.812464 10630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:09.814296 10637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f33265312a042c79002151963f04605. Latest consensus state: current_term: 1 leader_uuid: "6f33265312a042c79002151963f04605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f33265312a042c79002151963f04605" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38239 } } }
I20260807 02:22:09.814455 10637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:09.814934 10644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:09.814849 10636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f33265312a042c79002151963f04605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f33265312a042c79002151963f04605" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38239 } } }
I20260807 02:22:09.815018 10636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:09.818627 10644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:09.825425 10644 catalog_manager.cc:1347] Generated new cluster ID: 2f12618922e04fe78d63106a683b5c10
I20260807 02:22:09.825531 10644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:09.841784 10644 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:22:09.842878 10644 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:09.851198 10644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f33265312a042c79002151963f04605: Generated new TSK 0
I20260807 02:22:09.852054 10644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:22:09.999591 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:09.865627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.4.221.126:38239)
0807 02:22:09.865791 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:09.865796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:09.866024 (+ 228us) server_negotiation.cc:407] Connection header received
0807 02:22:09.866295 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:09.866298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:09.866353 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:09.866481 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0807 02:22:09.866760 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:09.867317 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:09.868292 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:09.868518 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:09.868641 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:09.869014 (+ 373us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0807 02:22:09.998997 (+129983us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 02:22:09.999415 (+ 418us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55132: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":70}
W20260807 02:22:09.999706 10633 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55132: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260807 02:22:09.999775 10660 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:09.865474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:38239 (local address 127.0.0.1:55132)
0807 02:22:09.865907 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:09.865929 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:09.866075 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:09.866483 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:09.866486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:09.866496 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0807 02:22:09.866623 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:09.866632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:09.867446 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:09.867450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:09.868141 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:09.868158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:09.868660 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:09.868664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:09.868769 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:09.868805 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0807 02:22:09.999313 (+130508us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0807 02:22:09.999536 (+ 223us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.221.126:38239: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":137,"threads_started":1}
I20260807 02:22:10.000571 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 10560
2026-08-07T02:22:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (571 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (149 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (135 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:22:10 dist-test-slave-69rg krb5kdc[10672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:22:10 dist-test-slave-69rg krb5kdc[10672](info): set up 2 sockets
Aug 07 02:22:10 dist-test-slave-69rg krb5kdc[10672](info): commencing operation
krb5kdc: starting...
W20260807 02:22:11.626338 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.318s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:22:11 dist-test-slave-69rg krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:22:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:22:11.766198 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:33909
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:33909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:11.885182 10686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:11.885514 10686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:11.885589 10686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:11.889654 10686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:11.889756 10686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:11.889797 10686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:11.889869 10686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:11.889925 10686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:22:11.895458 10686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:33909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:33909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:11.896708 10686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:11.897850 10686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:11.907569 10695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:11.907629 10691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:11.907594 10686 server_base.cc:1034] running on GCE node
W20260807 02:22:11.907725 10692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:11.908447 10686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:11.909648 10686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:11.910832 10686 hybrid_clock.cc:648] HybridClock initialized: now 1786069331910817 us; error 40 us; skew 500 ppm
Aug 07 02:22:11 dist-test-slave-69rg krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069331, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:11.919274 10686 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:22:11.920336 10686 webserver.cc:460] Webserver started at http://127.4.221.126:45617/ using document root <none> and password file <none>
I20260807 02:22:11.920926 10686 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:11.920995 10686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:11.921183 10686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:11.923435 10686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "fd8267abd0784b6095c7696f50eecad2"
format_stamp: "Formatted at 2026-08-07 02:22:11 on dist-test-slave-69rg"
server_key: "63b33396b2f383b00152a2befd3e54b3"
server_key_iv: "8e2f9b0f49eda5e916f83eeffb496969"
server_key_version: "encryptionkey@0"
I20260807 02:22:11.924072 10686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "fd8267abd0784b6095c7696f50eecad2"
format_stamp: "Formatted at 2026-08-07 02:22:11 on dist-test-slave-69rg"
server_key: "63b33396b2f383b00152a2befd3e54b3"
server_key_iv: "8e2f9b0f49eda5e916f83eeffb496969"
server_key_version: "encryptionkey@0"
I20260807 02:22:11.928145 10686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:22:11.930851 10703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:11.932204 10686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:22:11.932370 10686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "fd8267abd0784b6095c7696f50eecad2"
format_stamp: "Formatted at 2026-08-07 02:22:11 on dist-test-slave-69rg"
server_key: "63b33396b2f383b00152a2befd3e54b3"
server_key_iv: "8e2f9b0f49eda5e916f83eeffb496969"
server_key_version: "encryptionkey@0"
I20260807 02:22:11.932495 10686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:11.958492 10686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:11.962179 10686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:11.962456 10686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:11.970216 10686 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:33909
I20260807 02:22:11.970235 10757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:33909 every 8 connection(s)
I20260807 02:22:11.971278 10686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:11.974224 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:11.974812 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 10686
I20260807 02:22:11.974941 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:22:11.975278 4981 external_mini_cluster.cc:1428] Setting key 499919bc98d9a99a2b788894d7147e99
I20260807 02:22:11.981577 10758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2: Bootstrap starting.
I20260807 02:22:11.984149 10758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2: Neither blocks nor log segments found. Creating new log.
Aug 07 02:22:11 dist-test-slave-69rg krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:11.985071 10758 log.cc:824] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:11.987776 10758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2: No bootstrap required, opened a new log
I20260807 02:22:11.991175 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:11.976917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43636 (local address 127.4.221.126:33909)
0807 02:22:11.977407 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:11.977422 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:11.977460 (+ 38us) server_negotiation.cc:407] Connection header received
0807 02:22:11.978322 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:11.978342 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:11.978691 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:11.979054 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:11.979932 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:11.980960 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:11.981930 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:11.982429 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:11.982628 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:11.985553 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:11.985559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:11.985575 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:11.985604 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:11.987665 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:11.988222 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:11.988228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:11.988234 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:11.988383 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:11.988661 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:11.988665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:11.988666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:11.989207 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:11.989403 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:11.989676 (+ 273us) server_negotiation.cc:299] Negotiation successful
0807 02:22:11.990054 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":138,"threads_started":1}
I20260807 02:22:11.992764 10758 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8267abd0784b6095c7696f50eecad2" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 33909 } }
I20260807 02:22:11.992998 10758 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:11.993040 10758 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8267abd0784b6095c7696f50eecad2, State: Initialized, Role: FOLLOWER
I20260807 02:22:11.993603 10758 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8267abd0784b6095c7696f50eecad2" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 33909 } }
I20260807 02:22:11.993784 10758 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:11.993906 10758 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:11.994014 10758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:11.995078 10758 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8267abd0784b6095c7696f50eecad2" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 33909 } }
I20260807 02:22:11.995448 10758 leader_election.cc:302] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd8267abd0784b6095c7696f50eecad2; no voters:
I20260807 02:22:11.995798 10758 leader_election.cc:288] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:22:11.996014 10763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:11.996273 10763 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [term 1 LEADER]: Becoming Leader. State: Replica: fd8267abd0784b6095c7696f50eecad2, State: Running, Role: LEADER
I20260807 02:22:11.996610 10763 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8267abd0784b6095c7696f50eecad2" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 33909 } }
I20260807 02:22:11.997200 10758 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:11.998394 10765 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd8267abd0784b6095c7696f50eecad2. Latest consensus state: current_term: 1 leader_uuid: "fd8267abd0784b6095c7696f50eecad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8267abd0784b6095c7696f50eecad2" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 33909 } } }
I20260807 02:22:11.998711 10764 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd8267abd0784b6095c7696f50eecad2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8267abd0784b6095c7696f50eecad2" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 33909 } } }
I20260807 02:22:11.998834 10765 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:11.998917 10764 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:11.999357 10774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:12.002430 10774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:12.009649 10774 catalog_manager.cc:1347] Generated new cluster ID: 16283f627da949798c50402ce90f061d
I20260807 02:22:12.009766 10774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:12.033499 10774 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:22:12.034610 10774 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:12.046478 10774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd8267abd0784b6095c7696f50eecad2: Generated new TSK 0
I20260807 02:22:12.047333 10774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:22:12.082402 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:33909
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:12.202659 10784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:12.203007 10784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:12.203086 10784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:12.207304 10784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:12.207399 10784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:12.207496 10784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:22:12.213020 10784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:33909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:12.214320 10784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:12.215427 10784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:12.225630 10789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:12.225611 10790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:12.225811 10784 server_base.cc:1034] running on GCE node
W20260807 02:22:12.225687 10792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:12.226393 10784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:12.227025 10784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:12.228228 10784 hybrid_clock.cc:648] HybridClock initialized: now 1786069332228207 us; error 32 us; skew 500 ppm
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:12.231513 10784 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:22:12.232539 10784 webserver.cc:460] Webserver started at http://127.4.221.65:44509/ using document root <none> and password file <none>
I20260807 02:22:12.233084 10784 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:12.233155 10784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:12.233330 10784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:12.235492 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "d79802614e134c909d0ac677c5050648"
format_stamp: "Formatted at 2026-08-07 02:22:12 on dist-test-slave-69rg"
server_key: "11ad011bfcb2e1f77ef4b4779a68c8aa"
server_key_iv: "1b1a047b75053e5550094e5fa91b81b8"
server_key_version: "encryptionkey@0"
I20260807 02:22:12.236218 10784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "d79802614e134c909d0ac677c5050648"
format_stamp: "Formatted at 2026-08-07 02:22:12 on dist-test-slave-69rg"
server_key: "11ad011bfcb2e1f77ef4b4779a68c8aa"
server_key_iv: "1b1a047b75053e5550094e5fa91b81b8"
server_key_version: "encryptionkey@0"
I20260807 02:22:12.240499 10784 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260807 02:22:12.243331 10799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:12.244839 10784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:22:12.244985 10784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "d79802614e134c909d0ac677c5050648"
format_stamp: "Formatted at 2026-08-07 02:22:12 on dist-test-slave-69rg"
server_key: "11ad011bfcb2e1f77ef4b4779a68c8aa"
server_key_iv: "1b1a047b75053e5550094e5fa91b81b8"
server_key_version: "encryptionkey@0"
I20260807 02:22:12.245119 10784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:12.268221 10784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:12.271873 10784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:12.272115 10784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:12.272806 10784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:12.273775 10784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:12.273881 10784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:12.273963 10784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:12.273998 10784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:12.283763 10784 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:42403
I20260807 02:22:12.283821 10917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:42403 every 8 connection(s)
I20260807 02:22:12.284775 10784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:22:12.291335 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 10784
I20260807 02:22:12.291457 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:22:12.291771 4981 external_mini_cluster.cc:1428] Setting key 3b872b31d698cbdd54de9e5db042e280
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:12.301291 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:12.286884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:57755 (local address 127.4.221.126:33909)
0807 02:22:12.287041 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:12.287045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:12.287914 (+ 869us) server_negotiation.cc:407] Connection header received
0807 02:22:12.289526 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:12.289530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:12.289594 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:12.289717 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:12.291137 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.291792 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.293017 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.293264 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.293387 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:12.296573 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:12.296577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:12.296579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:12.296606 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:12.298742 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:12.299533 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:12.299537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:12.299539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:12.299635 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:12.300046 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:12.300049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:12.300051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:12.300281 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:12.300393 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:12.300968 (+ 575us) server_negotiation.cc:299] Negotiation successful
0807 02:22:12.301113 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 02:22:12.302287 10920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:12.287274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:33909 (local address 127.4.221.65:57755)
0807 02:22:12.287743 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:12.287790 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:12.289280 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:12.289932 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:12.289941 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:12.290351 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:12.290953 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:12.290968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.291949 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.291952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:12.292878 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:12.292887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.293440 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.293445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:12.293640 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:12.294453 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:12.294472 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:12.296335 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:12.298943 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:12.298953 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:12.298973 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:12.299382 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:12.299752 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:12.299755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:12.299757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:12.299933 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:12.300396 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:12.300406 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:12.300714 (+ 308us) client_negotiation.cc:769] Sending connection context
0807 02:22:12.300928 (+ 214us) client_negotiation.cc:240] Negotiation successful
0807 02:22:12.301309 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
I20260807 02:22:12.303668 10918 heartbeater.cc:347] Connected to a master server at 127.4.221.126:33909
I20260807 02:22:12.303952 10918 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:12.304561 10918 heartbeater.cc:511] Master 127.4.221.126:33909 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
I20260807 02:22:12.306318 10722 ts_manager.cc:194] Registered new tserver with Master: d79802614e134c909d0ac677c5050648 (127.4.221.65:42403)
I20260807 02:22:12.307924 10722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:57755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:22:12.326617 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:33909
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:12.450906 10923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:12.451270 10923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:12.451351 10923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:12.455665 10923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:12.455735 10923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:12.455799 10923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:22:12.461551 10923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:33909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:12.462843 10923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:12.463981 10923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:12.474380 10931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:12.474395 10929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:12.474380 10928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:12.474707 10923 server_base.cc:1034] running on GCE node
I20260807 02:22:12.475129 10923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:12.475836 10923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:12.477038 10923 hybrid_clock.cc:648] HybridClock initialized: now 1786069332476998 us; error 53 us; skew 500 ppm
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:12.480157 10923 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:22:12.481138 10923 webserver.cc:460] Webserver started at http://127.4.221.66:45573/ using document root <none> and password file <none>
I20260807 02:22:12.481778 10923 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:12.481935 10923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:12.482197 10923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:12.484429 10923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "81961c7ef3aa4d0499bfe3968fd5f665"
format_stamp: "Formatted at 2026-08-07 02:22:12 on dist-test-slave-69rg"
server_key: "d07c577d66499b04f28d404c8213fad7"
server_key_iv: "9392cf0a7835cf85007aae810e8084b8"
server_key_version: "encryptionkey@0"
I20260807 02:22:12.485059 10923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "81961c7ef3aa4d0499bfe3968fd5f665"
format_stamp: "Formatted at 2026-08-07 02:22:12 on dist-test-slave-69rg"
server_key: "d07c577d66499b04f28d404c8213fad7"
server_key_iv: "9392cf0a7835cf85007aae810e8084b8"
server_key_version: "encryptionkey@0"
I20260807 02:22:12.489303 10923 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 02:22:12.492398 10938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:12.494068 10923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:22:12.494236 10923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "81961c7ef3aa4d0499bfe3968fd5f665"
format_stamp: "Formatted at 2026-08-07 02:22:12 on dist-test-slave-69rg"
server_key: "d07c577d66499b04f28d404c8213fad7"
server_key_iv: "9392cf0a7835cf85007aae810e8084b8"
server_key_version: "encryptionkey@0"
I20260807 02:22:12.494375 10923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:12.507624 10923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:12.511382 10923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:12.511615 10923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:12.512280 10923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:12.513162 10923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:12.513235 10923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:12.513307 10923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:12.513358 10923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:12.522750 10923 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:43977
I20260807 02:22:12.522846 11055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:43977 every 8 connection(s)
I20260807 02:22:12.523705 10923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:22:12.524892 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 10923
I20260807 02:22:12.525022 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:22:12.525326 4981 external_mini_cluster.cc:1428] Setting key fa567d574c63b12ed8a76a66a839d0fd
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
I20260807 02:22:12.542372 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:12.525984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:51279 (local address 127.4.221.126:33909)
0807 02:22:12.526157 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:12.526162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:12.527173 (+ 1011us) server_negotiation.cc:407] Connection header received
0807 02:22:12.529395 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:12.529401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:12.529471 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:12.529579 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:12.531164 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.532139 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.533748 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.534272 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.534420 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:12.537664 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:12.537667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:12.537670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:12.537701 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:12.539561 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:12.540243 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:12.540246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:12.540247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:12.540406 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:12.540875 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:12.540878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:12.540880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:12.541186 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:12.541313 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:12.541909 (+ 596us) server_negotiation.cc:299] Negotiation successful
0807 02:22:12.542096 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 02:22:12.543167 11058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:12.526437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:33909 (local address 127.4.221.66:51279)
0807 02:22:12.526990 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:12.527054 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:12.529082 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:12.529952 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:12.529959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:12.530397 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:12.530950 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:12.530968 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.532353 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.532357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:12.533574 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:12.533588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.534499 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.534502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:12.534689 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:12.535457 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:12.535478 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:12.537447 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:12.539712 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:12.539718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:12.539733 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:12.540106 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:12.540515 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:12.540521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:12.540524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:12.540747 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:12.541326 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:12.541332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:12.541610 (+ 278us) client_negotiation.cc:769] Sending connection context
0807 02:22:12.541794 (+ 184us) client_negotiation.cc:240] Negotiation successful
0807 02:22:12.542168 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":168,"threads_started":1}
I20260807 02:22:12.544610 11056 heartbeater.cc:347] Connected to a master server at 127.4.221.126:33909
I20260807 02:22:12.544874 11056 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:12.545507 11056 heartbeater.cc:511] Master 127.4.221.126:33909 requested a full tablet report, sending...
I20260807 02:22:12.546823 10722 ts_manager.cc:194] Registered new tserver with Master: 81961c7ef3aa4d0499bfe3968fd5f665 (127.4.221.66:43977)
I20260807 02:22:12.547447 10722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:51279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:22:12.561964 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:33909
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:12.680970 11062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:12.681273 11062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:12.681319 11062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:12.685527 11062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:12.685613 11062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:12.685676 11062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:22:12.691454 11062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:33909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:12.692634 11062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:12.693671 11062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:12.703599 11070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:12.703583 11067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:12.703868 11062 server_base.cc:1034] running on GCE node
W20260807 02:22:12.703583 11068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:12.704329 11062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:12.704917 11062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:12.706076 11062 hybrid_clock.cc:648] HybridClock initialized: now 1786069332706063 us; error 24 us; skew 500 ppm
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:12.708935 11062 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:22:12.709919 11062 webserver.cc:460] Webserver started at http://127.4.221.67:37275/ using document root <none> and password file <none>
I20260807 02:22:12.710460 11062 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:12.710518 11062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:12.710690 11062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:12.712785 11062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "25ad08aa46da4481a7a52b082c7d4173"
format_stamp: "Formatted at 2026-08-07 02:22:12 on dist-test-slave-69rg"
server_key: "2268806fecb94a1f1a48db279e6843b0"
server_key_iv: "d444223c1a2327bfc5bff37972a93935"
server_key_version: "encryptionkey@0"
I20260807 02:22:12.713354 11062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "25ad08aa46da4481a7a52b082c7d4173"
format_stamp: "Formatted at 2026-08-07 02:22:12 on dist-test-slave-69rg"
server_key: "2268806fecb94a1f1a48db279e6843b0"
server_key_iv: "d444223c1a2327bfc5bff37972a93935"
server_key_version: "encryptionkey@0"
I20260807 02:22:12.717428 11062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260807 02:22:12.720386 11077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:12.721779 11062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:22:12.721994 11062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "25ad08aa46da4481a7a52b082c7d4173"
format_stamp: "Formatted at 2026-08-07 02:22:12 on dist-test-slave-69rg"
server_key: "2268806fecb94a1f1a48db279e6843b0"
server_key_iv: "d444223c1a2327bfc5bff37972a93935"
server_key_version: "encryptionkey@0"
I20260807 02:22:12.722126 11062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:12.753211 11062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:12.756955 11062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:12.757182 11062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:12.757899 11062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:12.758801 11062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:12.758847 11062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:12.758915 11062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:12.758951 11062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:12.768965 11062 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:40205
I20260807 02:22:12.768985 11194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:40205 every 8 connection(s)
I20260807 02:22:12.769963 11062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:22:12.770417 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 11062
I20260807 02:22:12.770550 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:22:12.770852 4981 external_mini_cluster.cc:1428] Setting key 0842aa45c69360353062f10db442699a
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:12.784546 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:12.771869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:43165 (local address 127.4.221.126:33909)
0807 02:22:12.772014 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:12.772018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:12.772997 (+ 979us) server_negotiation.cc:407] Connection header received
0807 02:22:12.774594 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:12.774598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:12.774654 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:12.774743 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:12.775923 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.776475 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.777655 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.777952 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.778064 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:12.780659 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:12.780663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:12.780665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:12.780709 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:12.782284 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:12.782835 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:12.782838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:12.782840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:12.782930 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:12.783311 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:12.783315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:12.783317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:12.783524 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:12.783635 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:12.784243 (+ 608us) server_negotiation.cc:299] Negotiation successful
0807 02:22:12.784382 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 02:22:12.785610 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:12.772343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:33909 (local address 127.4.221.67:43165)
0807 02:22:12.772850 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:12.772892 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:12.774384 (+ 1492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:12.774909 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:12.774915 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:12.775298 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:12.775776 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:12.775788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.776607 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.776615 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:12.777492 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:12.777501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.778105 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.778125 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:12.778304 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:12.778947 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:12.778963 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:12.780489 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:12.782424 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:12.782430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:12.782447 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:12.782724 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:12.783026 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:12.783030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:12.783032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:12.783206 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:12.783633 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:12.783640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:12.783993 (+ 353us) client_negotiation.cc:769] Sending connection context
0807 02:22:12.784205 (+ 212us) client_negotiation.cc:240] Negotiation successful
0807 02:22:12.784557 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":172,"threads_started":1}
I20260807 02:22:12.786870 11195 heartbeater.cc:347] Connected to a master server at 127.4.221.126:33909
I20260807 02:22:12.787135 11195 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:12.787645 11195 heartbeater.cc:511] Master 127.4.221.126:33909 requested a full tablet report, sending...
I20260807 02:22:12.788725 10722 ts_manager.cc:194] Registered new tserver with Master: 25ad08aa46da4481a7a52b082c7d4173 (127.4.221.67:40205)
I20260807 02:22:12.789283 10722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:43165
I20260807 02:22:12.797670 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/test-admin.keytab.
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:12.815832 4981 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260807 02:22:12.819065 11202 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:12.817366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.4.221.65:42403)
0807 02:22:12.817722 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:12.817729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:12.817751 (+ 22us) server_negotiation.cc:407] Connection header received
0807 02:22:12.817801 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:12.817806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:12.818056 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:12.818262 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:12.818879 (+ 617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55978: BlockingRecv error: recv got EOF from 127.0.0.1:55978 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
W20260807 02:22:12.819191 4981 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/krb5cc' of type FILE
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:12.820405 4981 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:12.821452 4981 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.65@KRBTEST.COM
I20260807 02:22:12.832628 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:12.822610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.4.221.65:42403)
0807 02:22:12.822771 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:12.822775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:12.822791 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:22:12.822914 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:12.822917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:12.822961 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:12.823033 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:12.823660 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.824476 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.825513 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.826143 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.826284 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:12.828987 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:12.828999 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:12.829015 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:12.829044 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:12.830769 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:12.831247 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:12.831253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:12.831254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:12.831341 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:12.831603 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:12.831605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:12.831606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:12.831873 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:12.832047 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:12.832265 (+ 218us) server_negotiation.cc:299] Negotiation successful
0807 02:22:12.832474 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260807 02:22:12.837226 11202 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:12.836009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.4.221.65:42403)
0807 02:22:12.836134 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:12.836137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:12.836298 (+ 161us) server_negotiation.cc:407] Connection header received
0807 02:22:12.836406 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:12.836409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:12.836455 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:12.836528 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:12.837109 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56004: BlockingRecv error: recv got EOF from 127.0.0.1:56004 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260807 02:22:12.837460 4981 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/krb5cc' of type FILE
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:12.838594 4981 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:12.839517 4981 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.65@KRBTEST.COM
I20260807 02:22:12.849436 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:12.840474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.4.221.65:42403)
0807 02:22:12.840633 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:12.840638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:12.840762 (+ 124us) server_negotiation.cc:407] Connection header received
0807 02:22:12.840926 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:12.840929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:12.840979 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:12.841068 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:12.841666 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.842293 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.843107 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.843376 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.843491 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:12.846066 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:12.846069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:12.846071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:12.846100 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:12.847558 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:12.848078 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:12.848082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:12.848084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:12.848176 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:12.848469 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:12.848473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:12.848475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:12.848727 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:12.848906 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:12.849135 (+ 229us) server_negotiation.cc:299] Negotiation successful
0807 02:22:12.849273 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260807 02:22:12.852607 11202 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:12.851521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.4.221.65:42403)
0807 02:22:12.851684 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:12.851687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:12.851716 (+ 29us) server_negotiation.cc:407] Connection header received
0807 02:22:12.851848 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:12.851850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:12.851891 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:12.851976 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:12.852505 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56028: BlockingRecv error: recv got EOF from 127.0.0.1:56028 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260807 02:22:12.852741 4981 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786069301823597-4981-0/krb5kdc/krb5cc' of type FILE
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:12.853967 4981 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:12.854907 4981 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[10672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.65@KRBTEST.COM
I20260807 02:22:12.864850 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:12.855840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.4.221.65:42403)
0807 02:22:12.855990 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:12.855995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:12.856118 (+ 123us) server_negotiation.cc:407] Connection header received
0807 02:22:12.856286 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:12.856289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:12.856342 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:12.856432 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:12.856963 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.857490 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.858390 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:12.858629 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:12.858742 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:12.861632 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:12.861636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:12.861639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:12.861676 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:12.863075 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:12.863545 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:12.863549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:12.863551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:12.863638 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:12.863938 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:12.863941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:12.863943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:12.864141 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:12.864314 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:12.864552 (+ 238us) server_negotiation.cc:299] Negotiation successful
0807 02:22:12.864675 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 02:22:12.866137 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 10784
I20260807 02:22:12.874598 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 10923
I20260807 02:22:12.881500 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 11062
I20260807 02:22:12.887893 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 10686
2026-08-07T02:22:12Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2603 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[11216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[11216](info): set up 2 sockets
Aug 07 02:22:12 dist-test-slave-69rg krb5kdc[11216](info): commencing operation
krb5kdc: starting...
W20260807 02:22:13.958324 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:22:14 dist-test-slave-69rg krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:22:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.126.keytab.
Entry for principal oryx/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.126.keytab.
I20260807 02:22:14.097606 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:36267
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.126.keytab
--principal=oryx/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:36267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:14.217216 11231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:14.217533 11231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:14.217579 11231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:14.221719 11231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:14.221798 11231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:14.221848 11231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:14.221866 11231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:14.221882 11231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:22:14.227596 11231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:36267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.126.keytab
--principal=oryx/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:36267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:14.229043 11231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:14.230211 11231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:14.239635 11239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:14.239620 11237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:14.239890 11231 server_base.cc:1034] running on GCE node
W20260807 02:22:14.239620 11236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:14.240732 11231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:14.241989 11231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:14.243211 11231 hybrid_clock.cc:648] HybridClock initialized: now 1786069334243183 us; error 43 us; skew 500 ppm
Aug 07 02:22:14 dist-test-slave-69rg krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069334, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:14.246268 11231 init.cc:377] Logged in from keytab as oryx/127.4.221.126@KRBTEST.COM (short username oryx)
I20260807 02:22:14.247212 11231 webserver.cc:460] Webserver started at http://127.4.221.126:45875/ using document root <none> and password file <none>
I20260807 02:22:14.247808 11231 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:14.247907 11231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:14.248186 11231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:14.250389 11231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "9fb3edcd97e443208276838abdc871af"
format_stamp: "Formatted at 2026-08-07 02:22:14 on dist-test-slave-69rg"
server_key: "fb9c8d82bee44edaebb774c986f4563d"
server_key_iv: "8efccefe476b498713e0954355ec9227"
server_key_version: "encryptionkey@0"
I20260807 02:22:14.250979 11231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "9fb3edcd97e443208276838abdc871af"
format_stamp: "Formatted at 2026-08-07 02:22:14 on dist-test-slave-69rg"
server_key: "fb9c8d82bee44edaebb774c986f4563d"
server_key_iv: "8efccefe476b498713e0954355ec9227"
server_key_version: "encryptionkey@0"
I20260807 02:22:14.254827 11231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 02:22:14.257352 11246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:14.258713 11231 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 02:22:14.258862 11231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "9fb3edcd97e443208276838abdc871af"
format_stamp: "Formatted at 2026-08-07 02:22:14 on dist-test-slave-69rg"
server_key: "fb9c8d82bee44edaebb774c986f4563d"
server_key_iv: "8efccefe476b498713e0954355ec9227"
server_key_version: "encryptionkey@0"
I20260807 02:22:14.258981 11231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:14.287557 11231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:14.291260 11231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:14.291486 11231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:14.299317 11231 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:36267
I20260807 02:22:14.299643 11301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:36267 every 8 connection(s)
I20260807 02:22:14.300366 11231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:14.303448 11302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:14.306384 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 11231
I20260807 02:22:14.306562 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:22:14.306874 4981 external_mini_cluster.cc:1428] Setting key d1b6a7a894ce64f0c19d5ee3acde7c17
I20260807 02:22:14.309720 11302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af: Bootstrap starting.
I20260807 02:22:14.312492 11302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:14.313527 11302 log.cc:824] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:14.316236 11302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af: No bootstrap required, opened a new log
Aug 07 02:22:14 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
I20260807 02:22:14.320894 11302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } }
I20260807 02:22:14.321220 11302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:14.321302 11302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb3edcd97e443208276838abdc871af, State: Initialized, Role: FOLLOWER
I20260807 02:22:14.321915 11302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } }
I20260807 02:22:14.322089 11302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:14.322170 11302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:14.322276 11302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:14.323312 11302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } }
I20260807 02:22:14.323717 11302 leader_election.cc:302] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fb3edcd97e443208276838abdc871af; no voters:
I20260807 02:22:14.324110 11302 leader_election.cc:288] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:22:14.324196 11309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:14.324411 11309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [term 1 LEADER]: Becoming Leader. State: Replica: 9fb3edcd97e443208276838abdc871af, State: Running, Role: LEADER
I20260807 02:22:14.324579 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:14.308618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.4.221.126:36267)
0807 02:22:14.309097 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:14.309113 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:14.309152 (+ 39us) server_negotiation.cc:407] Connection header received
0807 02:22:14.309779 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:14.309796 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:14.310207 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:14.310491 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:14.311503 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:14.312590 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:14.313891 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:14.314604 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:14.314726 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:14.317769 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:14.317777 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:14.317796 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:14.317852 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:14.320755 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:14.321542 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:14.321549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:14.321556 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:14.321720 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:14.322105 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:14.322110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:14.322112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:14.322663 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:14.322924 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:14.323206 (+ 282us) server_negotiation.cc:299] Negotiation successful
0807 02:22:14.323544 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":181,"threads_started":1}
I20260807 02:22:14.324750 11309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } }
I20260807 02:22:14.325541 11302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:14.327399 11310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } }
I20260807 02:22:14.327560 11310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:14.327521 11311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fb3edcd97e443208276838abdc871af. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } }
I20260807 02:22:14.328064 11311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:14.328120 11324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:14.331287 11324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:14.337118 11324 catalog_manager.cc:1347] Generated new cluster ID: c81dc73836e744b8bc84b1df20e2776b
I20260807 02:22:14.337208 11324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:14.353243 11324 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:22:14.354573 11324 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:14.364813 11324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af: Generated new TSK 0
I20260807 02:22:14.365682 11324 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.65.keytab.
Entry for principal oryx/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.65.keytab.
I20260807 02:22:14.406541 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:36267
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.65.keytab
--principal=oryx/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:14.526042 11330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:14.526351 11330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:14.526424 11330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:14.530758 11330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:14.530854 11330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:14.530949 11330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:22:14.536415 11330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.65.keytab
--principal=oryx/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:36267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:14.537640 11330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:14.538798 11330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:14.549453 11338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:14.549454 11336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:14.549454 11335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:14.549516 11330 server_base.cc:1034] running on GCE node
I20260807 02:22:14.550312 11330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:14.551013 11330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:14.552240 11330 hybrid_clock.cc:648] HybridClock initialized: now 1786069334552204 us; error 48 us; skew 500 ppm
Aug 07 02:22:14 dist-test-slave-69rg krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069334, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:14.555434 11330 init.cc:377] Logged in from keytab as oryx/127.4.221.65@KRBTEST.COM (short username oryx)
I20260807 02:22:14.556391 11330 webserver.cc:460] Webserver started at http://127.4.221.65:36709/ using document root <none> and password file <none>
I20260807 02:22:14.556984 11330 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:14.557080 11330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:14.557325 11330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:14.559562 11330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "6904574eb48242d5828b8f304bab2f7d"
format_stamp: "Formatted at 2026-08-07 02:22:14 on dist-test-slave-69rg"
server_key: "eedc965ab8ec3745607787c1ebe435ea"
server_key_iv: "1689b9e47a13d77aba96b8172ffe2cec"
server_key_version: "encryptionkey@0"
I20260807 02:22:14.560163 11330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "6904574eb48242d5828b8f304bab2f7d"
format_stamp: "Formatted at 2026-08-07 02:22:14 on dist-test-slave-69rg"
server_key: "eedc965ab8ec3745607787c1ebe435ea"
server_key_iv: "1689b9e47a13d77aba96b8172ffe2cec"
server_key_version: "encryptionkey@0"
I20260807 02:22:14.564378 11330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260807 02:22:14.566922 11345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:14.568181 11330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:22:14.568329 11330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "6904574eb48242d5828b8f304bab2f7d"
format_stamp: "Formatted at 2026-08-07 02:22:14 on dist-test-slave-69rg"
server_key: "eedc965ab8ec3745607787c1ebe435ea"
server_key_iv: "1689b9e47a13d77aba96b8172ffe2cec"
server_key_version: "encryptionkey@0"
I20260807 02:22:14.568450 11330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:14.597174 11330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:14.600714 11330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:14.600936 11330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:14.601568 11330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:14.602484 11330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:14.602558 11330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:14.602631 11330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:14.602686 11330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:14.612076 11330 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:33815
I20260807 02:22:14.612128 11463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:33815 every 8 connection(s)
I20260807 02:22:14.613032 11330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:22:14.615290 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 11330
I20260807 02:22:14.615444 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:22:14.615748 4981 external_mini_cluster.cc:1428] Setting key c4f6bc7092c61d6f4a5dadebc1ce1fc0
Aug 07 02:22:14 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069334, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.4.221.66@KRBTEST.COM; defaulting to no policy
I20260807 02:22:14.628567 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:14.615112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:41071 (local address 127.4.221.126:36267)
0807 02:22:14.615333 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:14.615339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:14.616316 (+ 977us) server_negotiation.cc:407] Connection header received
0807 02:22:14.617944 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:14.617948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:14.618001 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:14.618085 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:14.619311 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:14.619868 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:14.621045 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:14.621440 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:14.621566 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:14.624435 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:14.624441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:14.624448 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:14.624488 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:14.626355 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:14.626877 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:14.626882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:14.626887 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:14.626994 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:14.627406 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:14.627409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:14.627411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:14.627694 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:14.627799 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:14.628315 (+ 516us) server_negotiation.cc:299] Negotiation successful
0807 02:22:14.628433 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260807 02:22:14.629456 11466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:14.615564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.65:41071)
0807 02:22:14.616120 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:14.616204 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:14.617695 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:14.618285 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:14.618293 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:14.618659 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:14.619148 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:14.619161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:14.620002 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:14.620005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:14.620914 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:14.620923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:14.621612 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:14.621615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:14.621774 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:14.622478 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:14.622495 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:14.624244 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:14.626471 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:14.626476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:14.626487 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:14.626745 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:14.627105 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:14.627107 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:14.627109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:14.627253 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:14.627812 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:14.627818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:14.628065 (+ 247us) client_negotiation.cc:769] Sending connection context
0807 02:22:14.628250 (+ 185us) client_negotiation.cc:240] Negotiation successful
0807 02:22:14.628524 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":153,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.221.66@KRBTEST.COM" created.
I20260807 02:22:14.630923 11464 heartbeater.cc:347] Connected to a master server at 127.4.221.126:36267
I20260807 02:22:14.631251 11464 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:14.631893 11464 heartbeater.cc:511] Master 127.4.221.126:36267 requested a full tablet report, sending...
I20260807 02:22:14.633647 11263 ts_manager.cc:194] Registered new tserver with Master: 6904574eb48242d5828b8f304bab2f7d (127.4.221.65:33815)
I20260807 02:22:14.634835 11263 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:41071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.66.keytab.
Entry for principal oryx/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.66.keytab.
I20260807 02:22:14.649725 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:36267
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.66.keytab
--principal=oryx/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:14.766566 11469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:14.766891 11469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:14.766969 11469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:14.772897 11469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:14.773000 11469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:14.773108 11469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:22:14.778968 11469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.66.keytab
--principal=oryx/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:36267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:14.780202 11469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:14.781306 11469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:14.791488 11477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:14.791531 11469 server_base.cc:1034] running on GCE node
W20260807 02:22:14.791486 11475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:14.791486 11474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:14.792209 11469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:14.792920 11469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:14.794109 11469 hybrid_clock.cc:648] HybridClock initialized: now 1786069334794078 us; error 50 us; skew 500 ppm
Aug 07 02:22:14 dist-test-slave-69rg krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069334, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:14.797916 11469 init.cc:377] Logged in from keytab as oryx/127.4.221.66@KRBTEST.COM (short username oryx)
I20260807 02:22:14.798908 11469 webserver.cc:460] Webserver started at http://127.4.221.66:37139/ using document root <none> and password file <none>
I20260807 02:22:14.799521 11469 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:14.799621 11469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:14.799873 11469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:14.802222 11469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "468cf228cf7e4d65b52b154eed448fb2"
format_stamp: "Formatted at 2026-08-07 02:22:14 on dist-test-slave-69rg"
server_key: "0ea92df9e7a33a480588bd7f1d4732d3"
server_key_iv: "f817e4bf45c767978e945fa600826c2f"
server_key_version: "encryptionkey@0"
I20260807 02:22:14.802856 11469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "468cf228cf7e4d65b52b154eed448fb2"
format_stamp: "Formatted at 2026-08-07 02:22:14 on dist-test-slave-69rg"
server_key: "0ea92df9e7a33a480588bd7f1d4732d3"
server_key_iv: "f817e4bf45c767978e945fa600826c2f"
server_key_version: "encryptionkey@0"
I20260807 02:22:14.806959 11469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:22:14.809891 11484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:14.811383 11469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:22:14.811563 11469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "468cf228cf7e4d65b52b154eed448fb2"
format_stamp: "Formatted at 2026-08-07 02:22:14 on dist-test-slave-69rg"
server_key: "0ea92df9e7a33a480588bd7f1d4732d3"
server_key_iv: "f817e4bf45c767978e945fa600826c2f"
server_key_version: "encryptionkey@0"
I20260807 02:22:14.811710 11469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:14.827281 11469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:14.832420 11469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:14.832736 11469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:14.833621 11469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:14.834972 11469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:14.835067 11469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:14.835178 11469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:14.835232 11469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:14.847936 11469 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:40755
I20260807 02:22:14.847962 11606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:40755 every 8 connection(s)
I20260807 02:22:14.849460 11469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:22:14.858067 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 11469
I20260807 02:22:14.858251 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:22:14.858588 4981 external_mini_cluster.cc:1428] Setting key 248307d3cd8910622fa29755376d18f9
Aug 07 02:22:14 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069334, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
I20260807 02:22:14.868819 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:14.852201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:57569 (local address 127.4.221.126:36267)
0807 02:22:14.852435 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:14.852441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:14.854010 (+ 1569us) server_negotiation.cc:407] Connection header received
0807 02:22:14.855868 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:14.855873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:14.855946 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:14.856036 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:14.857741 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:14.858533 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:14.860119 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:14.860457 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:14.860582 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:14.863850 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:14.863857 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:14.863862 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:14.863903 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:14.865953 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:14.866608 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:14.866621 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:14.866626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:14.866744 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:14.867161 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:14.867167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:14.867172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:14.867521 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:14.867650 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:14.868344 (+ 694us) server_negotiation.cc:299] Negotiation successful
0807 02:22:14.868525 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 02:22:14.869794 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:14.852895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.66:57569)
0807 02:22:14.853686 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:14.853753 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:14.855545 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:14.856285 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:14.856295 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:14.856851 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:14.857518 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:14.857536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:14.858700 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:14.858705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:14.859989 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:14.860001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:14.860663 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:14.860668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:14.860869 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:14.861654 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:14.861674 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:14.863635 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:14.866099 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:14.866105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:14.866128 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:14.866467 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:14.866857 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:14.866859 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:14.866861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:14.867028 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:14.867674 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:14.867682 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:14.868034 (+ 352us) client_negotiation.cc:769] Sending connection context
0807 02:22:14.868247 (+ 213us) client_negotiation.cc:240] Negotiation successful
0807 02:22:14.868599 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":173,"threads_started":1}
I20260807 02:22:14.871377 11607 heartbeater.cc:347] Connected to a master server at 127.4.221.126:36267
I20260807 02:22:14.871724 11607 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:14.872594 11607 heartbeater.cc:511] Master 127.4.221.126:36267 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.4.221.67@KRBTEST.COM; defaulting to no policy
I20260807 02:22:14.874117 11263 ts_manager.cc:194] Registered new tserver with Master: 468cf228cf7e4d65b52b154eed448fb2 (127.4.221.66:40755)
I20260807 02:22:14.874850 11263 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:57569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.67.keytab.
Entry for principal oryx/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.67.keytab.
I20260807 02:22:14.897115 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:36267
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.67.keytab
--principal=oryx/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:15.033491 11612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:15.033978 11612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:15.034072 11612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:15.038673 11612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:15.038800 11612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:15.038913 11612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:22:15.045387 11612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.67.keytab
--principal=oryx/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:36267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:15.047091 11612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:15.048606 11612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:15.060900 11620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:15.060904 11617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:15.060881 11618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:15.061059 11612 server_base.cc:1034] running on GCE node
I20260807 02:22:15.061723 11612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:15.062646 11612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:15.063835 11612 hybrid_clock.cc:648] HybridClock initialized: now 1786069335063793 us; error 56 us; skew 500 ppm
Aug 07 02:22:15 dist-test-slave-69rg krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069335, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:15.067550 11612 init.cc:377] Logged in from keytab as oryx/127.4.221.67@KRBTEST.COM (short username oryx)
I20260807 02:22:15.068723 11612 webserver.cc:460] Webserver started at http://127.4.221.67:42221/ using document root <none> and password file <none>
I20260807 02:22:15.069464 11612 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:15.069581 11612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:15.069938 11612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:15.072603 11612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4"
format_stamp: "Formatted at 2026-08-07 02:22:15 on dist-test-slave-69rg"
server_key: "c16af7422313561d2c500cc30e3fafa8"
server_key_iv: "7233f929ce07bf70af7bb75620f12593"
server_key_version: "encryptionkey@0"
I20260807 02:22:15.073321 11612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4"
format_stamp: "Formatted at 2026-08-07 02:22:15 on dist-test-slave-69rg"
server_key: "c16af7422313561d2c500cc30e3fafa8"
server_key_iv: "7233f929ce07bf70af7bb75620f12593"
server_key_version: "encryptionkey@0"
I20260807 02:22:15.078660 11612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 02:22:15.082049 11627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:15.083786 11612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:22:15.083961 11612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4"
format_stamp: "Formatted at 2026-08-07 02:22:15 on dist-test-slave-69rg"
server_key: "c16af7422313561d2c500cc30e3fafa8"
server_key_iv: "7233f929ce07bf70af7bb75620f12593"
server_key_version: "encryptionkey@0"
I20260807 02:22:15.084117 11612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:15.107662 11612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:15.111933 11612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:15.112213 11612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:15.113041 11612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:15.114183 11612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:15.114260 11612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:15.114339 11612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:15.114389 11612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:15.125442 11612 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:37593
I20260807 02:22:15.125517 11744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:37593 every 8 connection(s)
I20260807 02:22:15.126999 11612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:22:15.127774 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 11612
I20260807 02:22:15.127941 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:22:15.128342 4981 external_mini_cluster.cc:1428] Setting key eb40dd6809397c37067a26e924158582
Aug 07 02:22:15 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069335, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
I20260807 02:22:15.147910 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.130337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:51175 (local address 127.4.221.126:36267)
0807 02:22:15.130570 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.130575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.131953 (+ 1378us) server_negotiation.cc:407] Connection header received
0807 02:22:15.133719 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.133722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.133790 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.133927 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.135953 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.136648 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.138411 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.138709 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.138824 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.142805 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.142810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.142813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.142858 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.144940 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.145726 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.145733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.145737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.145926 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.146472 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.146475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.146477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.146743 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.146839 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.147597 (+ 758us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.147749 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 02:22:15.149116 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.130949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.67:51175)
0807 02:22:15.131743 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.131832 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.133421 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.134221 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.134236 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.134915 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:15.135695 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.135724 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.136864 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.136868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.138217 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.138232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.138960 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.138967 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.139195 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.140195 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:15.140225 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:15.142510 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:15.145113 (+ 2603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.145124 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.145163 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.145578 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.146060 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.146064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.146065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.146307 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.146890 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:15.146909 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:15.147292 (+ 383us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.147532 (+ 240us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.147942 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":160,"threads_started":1}
I20260807 02:22:15.150744 11745 heartbeater.cc:347] Connected to a master server at 127.4.221.126:36267
I20260807 02:22:15.151103 11745 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:15.151832 11745 heartbeater.cc:511] Master 127.4.221.126:36267 requested a full tablet report, sending...
I20260807 02:22:15.153178 11263 ts_manager.cc:194] Registered new tserver with Master: 40bb7a72cf2343b6beb35d6ddd62a7c4 (127.4.221.67:37593)
I20260807 02:22:15.153997 11263 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:51175
I20260807 02:22:15.156610 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.221.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.221.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.221.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.125.keytab.
Entry for principal oryx/127.4.221.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.125.keytab.
I20260807 02:22:15.193747 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.125:45039
--webserver_interface=127.4.221.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.125.keytab
--principal=oryx/127.4.221.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:36267,127.4.221.125:45039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:15.332438 11750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:15.332836 11750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:15.332887 11750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:15.337379 11750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:15.337484 11750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:15.337517 11750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:15.337543 11750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:15.337563 11750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:22:15.343724 11750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:36267,127.4.221.125:45039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.125.keytab
--principal=oryx/127.4.221.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.125:45039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.221.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
W20260807 02:22:15.344067 11750 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.221.126:36267,127.4.221.125:45039'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260807 02:22:15.345315 11750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:15.347239 11750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:15.358016 11758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:15.358016 11756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:15.358016 11755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:15.358206 11750 server_base.cc:1034] running on GCE node
I20260807 02:22:15.359186 11750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:15.360697 11750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:15.361948 11750 hybrid_clock.cc:648] HybridClock initialized: now 1786069335361910 us; error 56 us; skew 500 ppm
Aug 07 02:22:15 dist-test-slave-69rg krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069335, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:15.365980 11750 init.cc:377] Logged in from keytab as oryx/127.4.221.125@KRBTEST.COM (short username oryx)
I20260807 02:22:15.367240 11750 webserver.cc:460] Webserver started at http://127.4.221.125:37747/ using document root <none> and password file <none>
I20260807 02:22:15.368038 11750 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:15.368158 11750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:15.368464 11750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:15.371265 11750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/data/instance:
uuid: "54dffd44b40f425fb6309c0fa9618566"
format_stamp: "Formatted at 2026-08-07 02:22:15 on dist-test-slave-69rg"
server_key: "86cea794290b9a7264d6de8918142d32"
server_key_iv: "d7fc7c448bea1bf4352ea0b7ad817cbf"
server_key_version: "encryptionkey@0"
I20260807 02:22:15.372011 11750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/wal/instance:
uuid: "54dffd44b40f425fb6309c0fa9618566"
format_stamp: "Formatted at 2026-08-07 02:22:15 on dist-test-slave-69rg"
server_key: "86cea794290b9a7264d6de8918142d32"
server_key_iv: "d7fc7c448bea1bf4352ea0b7ad817cbf"
server_key_version: "encryptionkey@0"
I20260807 02:22:15.376935 11750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260807 02:22:15.380116 11766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:15.381711 11750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:22:15.381891 11750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/wal
uuid: "54dffd44b40f425fb6309c0fa9618566"
format_stamp: "Formatted at 2026-08-07 02:22:15 on dist-test-slave-69rg"
server_key: "86cea794290b9a7264d6de8918142d32"
server_key_iv: "d7fc7c448bea1bf4352ea0b7ad817cbf"
server_key_version: "encryptionkey@0"
I20260807 02:22:15.382027 11750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:15.395893 11750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:15.399940 11750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:15.400188 11750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:15.409247 11750 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.125:45039
I20260807 02:22:15.409757 11821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.125:45039 every 8 connection(s)
I20260807 02:22:15.410931 11750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/data/info.pb
I20260807 02:22:15.414084 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 11750
I20260807 02:22:15.414832 11822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:15.420686 11822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:22:15 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.221.125@KRBTEST.COM
Aug 07 02:22:15 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069335, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.125@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
I20260807 02:22:15.432847 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.416116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60336 (local address 127.4.221.125:45039)
0807 02:22:15.416694 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.416726 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.416773 (+ 47us) server_negotiation.cc:407] Connection header received
0807 02:22:15.417585 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.417605 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.418090 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.418456 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.419623 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.420861 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.422150 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.422704 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.422853 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.426086 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.426093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.426107 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.426143 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.428836 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.429420 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.429426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.429432 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.429572 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.429989 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.429993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.429994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.430623 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.430818 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.431425 (+ 607us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.431806 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":207,"threads_started":1}
I20260807 02:22:15.434580 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.423061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:42970)
0807 02:22:15.423621 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.423688 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.423826 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.424199 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.424205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.424628 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:15.424776 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.424782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.425805 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.425857 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.426949 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.426965 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.427685 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.427690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.427833 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.428397 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:15.428417 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:15.430503 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:15.432466 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.432477 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.432484 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.432733 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.433267 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.433270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.433272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.433371 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.433899 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:15.433904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:15.434041 (+ 137us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.434202 (+ 161us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.434410 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":206,"threads_started":1}
I20260807 02:22:15.434705 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.423181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.4.221.126:36267)
0807 02:22:15.423900 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.423907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.423926 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:22:15.423993 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.423996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.424044 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.424141 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.424936 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.425657 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.427107 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.427550 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.427641 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.430694 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.430701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.430706 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.430746 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.432327 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.432847 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.432850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.432852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.432947 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.433485 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.433488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.433489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.433759 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.433911 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.434345 (+ 434us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.434538 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598}
I20260807 02:22:15.436996 11822 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:22:15 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069335, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.125@KRBTEST.COM for oryx/127.4.221.125@KRBTEST.COM
I20260807 02:22:15.447899 11826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.437730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45039 (local address 127.0.0.1:60340)
0807 02:22:15.437914 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.437930 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.438015 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.438257 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.438259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.438442 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:15.438589 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.438594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.440041 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.440045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.440660 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.440668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.441148 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.441151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.441246 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.442036 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:15.442050 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:15.443718 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:15.446127 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.446130 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.446133 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.446375 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.446744 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.446747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.446749 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.446857 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.447334 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:15.447337 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:15.447430 (+ 93us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.447549 (+ 119us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.447702 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 02:22:15.447901 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.437850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60340 (local address 127.4.221.125:45039)
0807 02:22:15.437974 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.437982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.438000 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:22:15.438100 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.438102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.438153 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.438223 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.438724 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.439912 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.440790 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.441039 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.441126 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.443897 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.443901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.443903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.443930 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.445983 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.446514 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.446518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.446519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.446606 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.446969 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.446971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.446972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.447209 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.447298 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.447655 (+ 357us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.447776 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 02:22:15.452054 11822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: Bootstrap starting.
I20260807 02:22:15.455243 11822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:15.456163 11822 log.cc:824] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:15.458735 11822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: No bootstrap required, opened a new log
I20260807 02:22:15.462360 11822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } }
I20260807 02:22:15.462705 11822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:15.462801 11822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54dffd44b40f425fb6309c0fa9618566, State: Initialized, Role: FOLLOWER
I20260807 02:22:15.463459 11822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } }
I20260807 02:22:15.464083 11828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } } }
I20260807 02:22:15.464213 11828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:22:15.465014 11822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: configured and running, proceeding with master startup.
W20260807 02:22:15.470894 11839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:22:15.471004 11839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:22:15.473964 11750 master_runner.cc:428] Detected that this master 54dffd44b40f425fb6309c0fa9618566 is joining an existing cluster
I20260807 02:22:15.474047 11750 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.221.125:45039
I20260807 02:22:15.486052 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.477207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.4.221.125:45039)
0807 02:22:15.477309 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.477314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.477327 (+ 13us) server_negotiation.cc:407] Connection header received
0807 02:22:15.477363 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.477366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.477420 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.477516 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.478422 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.479249 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.480404 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.480729 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.480800 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.482304 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.482309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.482312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.482338 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.484108 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.484616 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.484620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.484622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.484711 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.485040 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.485045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.485048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.485254 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.485367 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.485642 (+ 275us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.485858 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":31}
I20260807 02:22:15.486446 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.477076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.4.221.126:36267)
0807 02:22:15.477358 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.477363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.477377 (+ 14us) server_negotiation.cc:407] Connection header received
0807 02:22:15.477424 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.477427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.477485 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.477577 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.478438 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.479062 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.480036 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.480304 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.480446 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.482461 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.482465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.482468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.482498 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.484388 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.484910 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.484914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.484916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.485019 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.485323 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.485326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.485328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.485559 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.485630 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.486074 (+ 444us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.486265 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260807 02:22:15.486068 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.476631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45039 (local address 127.0.0.1:60348)
0807 02:22:15.477091 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.477105 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.477203 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.477568 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.477571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.478035 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:15.478293 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.478302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.479386 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.479389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.480279 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.480289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.480944 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.480948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.481047 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.481551 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:15.481568 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:15.482145 (+ 577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:15.484248 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.484251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.484254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.484507 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.484815 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.484819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.484822 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.484925 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.485365 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:15.485368 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:15.485468 (+ 100us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.485582 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.485762 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":7,"thread_start_us":135,"threads_started":1}
I20260807 02:22:15.486932 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.476588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:42984)
0807 02:22:15.476937 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.476955 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.477093 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.477717 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.477719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.478075 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:15.478287 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.478294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.479191 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.479197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.479889 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.479898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.480449 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.480453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.480567 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.481794 (+ 1227us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:15.481808 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:15.482308 (+ 500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:15.484504 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.484507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.484510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.484738 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.485130 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.485133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.485135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.485221 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.485702 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:15.485705 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:15.485789 (+ 84us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.486599 (+ 810us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.486751 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":51,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for oryx/127.4.221.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.221.124@KRBTEST.COM" created.
Aug 07 02:22:15 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069334, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.126@KRBTEST.COM for oryx/127.4.221.125@KRBTEST.COM
I20260807 02:22:15.510075 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.494704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.4.221.125:45039)
0807 02:22:15.494855 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.494863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.494883 (+ 20us) server_negotiation.cc:407] Connection header received
0807 02:22:15.494972 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.494978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.495067 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.495173 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.496143 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.497164 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.498636 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.499055 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.499196 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.505383 (+ 6187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.505392 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.505399 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.505448 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.507435 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.508104 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.508107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.508109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.508193 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.508596 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.508600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.508602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.508971 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.509109 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.509664 (+ 555us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.509858 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 02:22:15.510273 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.493700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45039 (local address 127.0.0.1:60352)
0807 02:22:15.494260 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.494334 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.494484 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.495235 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.495243 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.495796 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:15.495972 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.495981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.497299 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.497302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.498487 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.498500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.499195 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.499199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.499330 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.500070 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:15.500086 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:15.505136 (+ 5050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:15.507598 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.507615 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.507625 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.507958 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.508305 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.508309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.508311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.508454 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.509142 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:15.509155 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:15.509365 (+ 210us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.509569 (+ 204us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.509924 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":180,"threads_started":1}
I20260807 02:22:15.512571 11263 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.221.125:45039: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "54dffd44b40f425fb6309c0fa9618566"
member_type: NON_VOTER
last_known_addr {
host: "127.4.221.125"
port: 45039
}
attrs {
promote: true
}
}
dest_uuid: "9fb3edcd97e443208276838abdc871af"
cas_config_opid_index: -1
I20260807 02:22:15.513579 11263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: NON_VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: true } }
I20260807 02:22:15.516733 11852 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 54dffd44b40f425fb6309c0fa9618566 (127.4.221.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: NON_VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: true } } }
I20260807 02:22:15.517638 11794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:15.518738 11852 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.221.125:45039
I20260807 02:22:15.519270 11750 master.cc:537] Master@127.4.221.125:45039 shutting down...
I20260807 02:22:15.519325 11851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: NON_VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: true } } }
I20260807 02:22:15.519443 11851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:15.519434 11794 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 FOLLOWER]: Refusing update from remote peer 9fb3edcd97e443208276838abdc871af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260807 02:22:15.520313 11851 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: NON_VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 02:22:15.523065 11247 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260807 02:22:15.524250 11247 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af -> Peer 54dffd44b40f425fb6309c0fa9618566 (127.4.221.125:45039): Couldn't send request to peer 54dffd44b40f425fb6309c0fa9618566. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 02:22:15.525494 11750 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:15.525957 11750 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:15.526116 11750 tablet_replica.cc:330] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.221.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.124.keytab.
Entry for principal oryx/127.4.221.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.124.keytab.
I20260807 02:22:15.529749 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.124:39679
--webserver_interface=127.4.221.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.124.keytab
--principal=oryx/127.4.221.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:36267,127.4.221.125:45039,127.4.221.124:39679
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:22:15.543898 11750 master.cc:552] Master@127.4.221.125:45039 shutdown complete.
I20260807 02:22:15.544004 11750 master_runner.cc:305] Clearing existing system tablet
I20260807 02:22:15.544327 11750 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:15.544396 11750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:15.546105 11750 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.003s sys 0.001s
I20260807 02:22:15.547325 11857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:15.547639 11750 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:15.547791 11750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 02:22:15.547837 11750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/wal
uuid: "54dffd44b40f425fb6309c0fa9618566"
format_stamp: "Formatted at 2026-08-07 02:22:15 on dist-test-slave-69rg"
server_key: "86cea794290b9a7264d6de8918142d32"
server_key_iv: "d7fc7c448bea1bf4352ea0b7ad817cbf"
server_key_version: "encryptionkey@0"
I20260807 02:22:15.550879 11750 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:15.554281 11750 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 02:22:15.554409 11750 log.cc:1196] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260807 02:22:15.554735 11750 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: Deleting consensus metadata
I20260807 02:22:15.554970 11750 master_runner.cc:315] Copying system tablet from 127.4.221.126:36267
I20260807 02:22:15.556945 11750 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: tablet copy: Beginning tablet copy session from remote peer at address 127.4.221.126:36267
I20260807 02:22:15.568197 11863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.557789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:42998)
0807 02:22:15.558377 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.558392 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.558542 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.558889 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.558892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.559160 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:15.559331 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.559339 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.560558 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.560563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.561527 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.561540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.562269 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.562273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.562399 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.563106 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:15.563122 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:15.563678 (+ 556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:15.566264 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.566268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.566271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.566545 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.566874 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.566877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.566880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.566980 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.567528 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:15.567532 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:15.567648 (+ 116us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.567761 (+ 113us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.567932 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":100,"threads_started":1}
I20260807 02:22:15.568188 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.558005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.4.221.126:36267)
0807 02:22:15.558215 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.558221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.558500 (+ 279us) server_negotiation.cc:407] Connection header received
0807 02:22:15.558651 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.558654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.558722 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.558847 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.559485 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.560414 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.561668 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.562102 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.562242 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.563841 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.563846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.563850 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.563882 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.566097 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.566658 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.566662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.566664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.566769 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.567131 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.567135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.567137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.567407 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.567530 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.567846 (+ 316us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.567998 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 02:22:15.569491 11285 tablet_copy_service.cc:140] P 9fb3edcd97e443208276838abdc871af: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 54dffd44b40f425fb6309c0fa9618566 ({username='oryx', principal='oryx/127.4.221.125@KRBTEST.COM'} at 127.0.0.1:42998)
I20260807 02:22:15.569726 11285 tablet_copy_service.cc:161] P 9fb3edcd97e443208276838abdc871af: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 54dffd44b40f425fb6309c0fa9618566 at {username='oryx', principal='oryx/127.4.221.125@KRBTEST.COM'} at 127.0.0.1:42998: session id = 54dffd44b40f425fb6309c0fa9618566-00000000000000000000000000000000
I20260807 02:22:15.572118 11285 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 02:22:15.574407 11750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 02:22:15.578086 11750 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: tablet copy: Starting download of 0 data blocks...
I20260807 02:22:15.578595 11750 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: tablet copy: Starting download of 1 WAL segments...
I20260807 02:22:15.581367 11750 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 02:22:15.584072 11285 tablet_copy_service.cc:342] P 9fb3edcd97e443208276838abdc871af: Request end of tablet copy session 54dffd44b40f425fb6309c0fa9618566-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.221.125@KRBTEST.COM'} at 127.0.0.1:42998
I20260807 02:22:15.584296 11285 tablet_copy_service.cc:433] P 9fb3edcd97e443208276838abdc871af: ending tablet copy session 54dffd44b40f425fb6309c0fa9618566-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 54dffd44b40f425fb6309c0fa9618566
I20260807 02:22:15.594677 11750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:15.596876 11868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:15.596896 11864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:15.597576 11750 server_base.cc:1034] running on GCE node
W20260807 02:22:15.598011 11865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:15.598275 11750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:15.598657 11750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:15.599812 11750 hybrid_clock.cc:648] HybridClock initialized: now 1786069335599797 us; error 42 us; skew 500 ppm
Aug 07 02:22:15 dist-test-slave-69rg krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069335, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:15.601931 11750 init.cc:377] Logged in from keytab as oryx/127.4.221.125@KRBTEST.COM (short username oryx)
I20260807 02:22:15.602543 11750 webserver.cc:460] Webserver started at http://127.4.221.125:35693/ using document root <none> and password file <none>
I20260807 02:22:15.602778 11750 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:15.602828 11750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:15.604610 11750 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260807 02:22:15.606088 11876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:15.606423 11750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260807 02:22:15.606510 11750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/wal
uuid: "54dffd44b40f425fb6309c0fa9618566"
format_stamp: "Formatted at 2026-08-07 02:22:15 on dist-test-slave-69rg"
server_key: "86cea794290b9a7264d6de8918142d32"
server_key_iv: "d7fc7c448bea1bf4352ea0b7ad817cbf"
server_key_version: "encryptionkey@0"
I20260807 02:22:15.606613 11750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:15.619375 11750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:15.622117 11750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:15.628657 11750 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.125:45039
I20260807 02:22:15.629458 11750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-1/data/info.pb
I20260807 02:22:15.629971 11930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.125:45039 every 8 connection(s)
I20260807 02:22:15.632082 11931 sys_catalog.cc:263] Verifying existing consensus state
I20260807 02:22:15.633481 11931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: Bootstrap starting.
I20260807 02:22:15.637413 11464 heartbeater.cc:503] Master 127.4.221.126:36267 was elected leader, sending a full tablet report...
I20260807 02:22:15.648754 11931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:22:15.649516 11931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: Bootstrap complete.
I20260807 02:22:15.650517 11931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: NON_VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: true } }
I20260807 02:22:15.650926 11931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 54dffd44b40f425fb6309c0fa9618566, State: Initialized, Role: LEARNER
I20260807 02:22:15.651168 11931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: NON_VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: true } }
I20260807 02:22:15.651759 11934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: NON_VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: true } } }
I20260807 02:22:15.651891 11934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: This master's current role is: LEARNER
I20260807 02:22:15.652016 11931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:15.656378 11946 catalog_manager.cc:1259] Loaded cluster ID: c81dc73836e744b8bc84b1df20e2776b
I20260807 02:22:15.656450 11946 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: loading cluster ID for follower catalog manager: success
I20260807 02:22:15.658958 11946 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: acquiring CA information for follower catalog manager: success
I20260807 02:22:15.660753 11946 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 02:22:15.710137 11855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:15.710595 11855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:15.710690 11855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:15.715799 11855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:15.715936 11855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:15.715975 11855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:15.716013 11855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:15.716074 11855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:22:15.722649 11855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:36267,127.4.221.125:45039,127.4.221.124:39679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.124.keytab
--principal=oryx/127.4.221.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.124:39679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.221.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:15.724339 11855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:15.725907 11855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 02:22:15.737641 11855 server_base.cc:1034] running on GCE node
W20260807 02:22:15.737514 11952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:15.737540 11954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:15.737509 11951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:15.738842 11855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:15.740594 11855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:15.741873 11855 hybrid_clock.cc:648] HybridClock initialized: now 1786069335741815 us; error 38 us; skew 500 ppm
Aug 07 02:22:15 dist-test-slave-69rg krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069335, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:15.745559 11855 init.cc:377] Logged in from keytab as oryx/127.4.221.124@KRBTEST.COM (short username oryx)
I20260807 02:22:15.746954 11855 webserver.cc:460] Webserver started at http://127.4.221.124:33259/ using document root <none> and password file <none>
I20260807 02:22:15.747748 11855 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:15.747859 11855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:15.748162 11855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:15.751040 11855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/data/instance:
uuid: "c332ab71f4fb4010ac997b04a494124f"
format_stamp: "Formatted at 2026-08-07 02:22:15 on dist-test-slave-69rg"
server_key: "20cd91862903e502764996353df05ad0"
server_key_iv: "65e32eae9a1f14e6b0c5dbaf29b6de3a"
server_key_version: "encryptionkey@0"
I20260807 02:22:15.751868 11855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/wal/instance:
uuid: "c332ab71f4fb4010ac997b04a494124f"
format_stamp: "Formatted at 2026-08-07 02:22:15 on dist-test-slave-69rg"
server_key: "20cd91862903e502764996353df05ad0"
server_key_iv: "65e32eae9a1f14e6b0c5dbaf29b6de3a"
server_key_version: "encryptionkey@0"
I20260807 02:22:15.757736 11855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260807 02:22:15.761324 11961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:15.763300 11855 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 02:22:15.763499 11855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/wal
uuid: "c332ab71f4fb4010ac997b04a494124f"
format_stamp: "Formatted at 2026-08-07 02:22:15 on dist-test-slave-69rg"
server_key: "20cd91862903e502764996353df05ad0"
server_key_iv: "65e32eae9a1f14e6b0c5dbaf29b6de3a"
server_key_version: "encryptionkey@0"
I20260807 02:22:15.763666 11855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:15.777941 11855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:15.782596 11855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:15.782873 11855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:15.792536 11855 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.124:39679
I20260807 02:22:15.792888 12014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.124:39679 every 8 connection(s)
I20260807 02:22:15.794202 11855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/data/info.pb
I20260807 02:22:15.798290 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:15.802210 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 11855
I20260807 02:22:15.803658 12015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:22:15 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069335, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.124@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
Aug 07 02:22:15 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.221.124@KRBTEST.COM
I20260807 02:22:15.818207 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.806355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.4.221.126:36267)
0807 02:22:15.806525 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.806530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.806753 (+ 223us) server_negotiation.cc:407] Connection header received
0807 02:22:15.806986 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.806989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.807049 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.807106 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.808242 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.808823 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.810010 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.810255 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.810382 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.813619 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.813623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.813626 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.813659 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.815404 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.816129 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.816134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.816136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.816239 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.816621 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.816628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.816630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.816957 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.817071 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.817762 (+ 691us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.817976 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 02:22:15.819610 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.804255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42438 (local address 127.4.221.124:39679)
0807 02:22:15.804820 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.804843 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.804896 (+ 53us) server_negotiation.cc:407] Connection header received
0807 02:22:15.805883 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.805910 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.806465 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.806920 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.807990 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.808992 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.809950 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.810360 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.810564 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.813921 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.813929 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.813943 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.813967 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.816081 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.816568 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.816577 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.816584 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.816662 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.817062 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.817066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.817068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.817482 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.817751 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.818027 (+ 276us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.818653 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":388,"thread_start_us":182,"threads_started":1}
I20260807 02:22:15.819610 12018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.806096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:43002)
0807 02:22:15.806581 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.806669 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.806854 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.807334 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.807340 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.807735 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:15.808084 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.808098 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.808995 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.808998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.809807 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.809868 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.810428 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.810432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.810566 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.811357 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:15.811376 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:15.813405 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:15.815622 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.815633 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.815650 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.816016 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.816339 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.816344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.816346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.816515 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.817081 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:15.817090 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:15.817495 (+ 405us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.817699 (+ 204us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.818242 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"spinlock_wait_cycles":12288,"thread_start_us":183,"threads_started":1}
I20260807 02:22:15.821277 12015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:22:15 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069335, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.124@KRBTEST.COM for oryx/127.4.221.125@KRBTEST.COM
I20260807 02:22:15.831616 12019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.822186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.4.221.125:45039)
0807 02:22:15.822598 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.822604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.822620 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:22:15.822674 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.822677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.822739 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.822832 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.823346 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.824147 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.825099 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.825349 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.825532 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.828174 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.828178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.828182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.828212 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.829726 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.830240 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.830247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.830249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.830347 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.830648 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.830653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.830655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.830877 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.831004 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.831282 (+ 278us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.831432 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":154,"threads_started":1}
I20260807 02:22:15.831614 12018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.822019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45039 (local address 127.0.0.1:60360)
0807 02:22:15.822149 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.822164 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.822247 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.822873 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.822876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.823066 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:15.823190 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.823195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.824270 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.824273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.824968 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.824978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.825531 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.825534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.825625 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.826245 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:15.826258 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:15.828006 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:15.829915 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.829919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.829921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.830130 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.830450 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.830453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.830456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.830547 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.830996 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:15.830999 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:15.831112 (+ 113us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.831224 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.831392 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260807 02:22:15.832676 12015 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:22:15 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069335, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.124@KRBTEST.COM for oryx/127.4.221.124@KRBTEST.COM
I20260807 02:22:15.841450 12018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.833362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:39679 (local address 127.0.0.1:42440)
0807 02:22:15.833512 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.833527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.833615 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.833919 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.833921 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.834083 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:15.834169 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.834175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.834922 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.834925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.835515 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.835524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.835980 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.835983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.836047 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.836556 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:15.836568 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:15.838082 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:15.839943 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.839946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.839947 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.840167 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.840423 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.840425 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.840426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.840502 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.840971 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:15.840973 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:15.841069 (+ 96us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.841153 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.841275 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 02:22:15.841462 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.833539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.4.221.124:39679)
0807 02:22:15.833654 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.833657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.833668 (+ 11us) server_negotiation.cc:407] Connection header received
0807 02:22:15.833696 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.833698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.833734 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.833800 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.834268 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.834801 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.835623 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.835879 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.835963 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.838233 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.838236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.838238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.838260 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.839829 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.840259 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.840262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.840263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.840340 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.840620 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.840624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.840625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.840862 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.840977 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.841221 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.841335 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 02:22:15.845973 12015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: Bootstrap starting.
I20260807 02:22:15.849785 12015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:15.850862 12015 log.cc:824] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:15.854200 12015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: No bootstrap required, opened a new log
I20260807 02:22:15.857869 12015 raft_consensus.cc:348] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } }
I20260807 02:22:15.858225 12015 raft_consensus.cc:374] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:15.858325 12015 raft_consensus.cc:729] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c332ab71f4fb4010ac997b04a494124f, State: Initialized, Role: FOLLOWER
I20260807 02:22:15.858907 12015 consensus_queue.cc:260] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } }
I20260807 02:22:15.859537 12021 sys_catalog.cc:455] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } } }
I20260807 02:22:15.859700 12021 sys_catalog.cc:458] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:22:15.860500 12015 sys_catalog.cc:564] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [sys.catalog]: configured and running, proceeding with master startup.
W20260807 02:22:15.866668 12033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:22:15.866782 12033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:22:15.874351 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 11330
I20260807 02:22:15.877902 11607 heartbeater.cc:503] Master 127.4.221.126:36267 was elected leader, sending a full tablet report...
I20260807 02:22:15.879274 11855 master_runner.cc:428] Detected that this master c332ab71f4fb4010ac997b04a494124f is joining an existing cluster
I20260807 02:22:15.879348 11855 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.221.124:39679
I20260807 02:22:15.885432 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:33815
--local_ip_for_outbound_sockets=127.4.221.65
--tserver_master_addrs=127.4.221.126:36267
--webserver_port=36709
--webserver_interface=127.4.221.65
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.65.keytab
--principal=oryx/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.221.126:36267,127.4.221.125:45039
--tserver_master_addrs=127.4.221.126:36267,127.4.221.125:45039,127.4.221.124:39679 with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:22:15.913425 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.882634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43016 (local address 127.4.221.126:36267)
0807 02:22:15.882781 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.882786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.882802 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:22:15.882853 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.882856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.882908 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.882997 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.885355 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.886586 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.887926 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.888336 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.888428 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.907293 (+ 18865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.907299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.907302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.907345 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.909647 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.911858 (+ 2211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.911864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.911866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.911969 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.912282 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.912286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.912287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.912553 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.912628 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.913092 (+ 464us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.913238 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:22:15.913666 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.883934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42444 (local address 127.4.221.124:39679)
0807 02:22:15.884030 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.884034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.889277 (+ 5243us) server_negotiation.cc:407] Connection header received
0807 02:22:15.889562 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.889565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.889614 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.889696 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.890326 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.891087 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.892304 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.892635 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.892746 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.909141 (+ 16395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.909147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.909152 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.909190 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.912038 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.912479 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.912483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.912484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.912553 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.912827 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.912830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.912831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.913027 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.913106 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.913380 (+ 274us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.913501 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260807 02:22:15.914206 12040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.883878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:39679 (local address 127.0.0.1:42444)
0807 02:22:15.889190 (+ 5312us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.889208 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.889460 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.889776 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.889782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.890060 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:15.890195 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.890202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.891218 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.891222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.892165 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.892179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.892770 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.892777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.892897 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.893448 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:15.893469 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:15.908739 (+ 15270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:15.912158 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.912161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.912163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.912384 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.912639 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.912641 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.912643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.912743 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.913147 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:15.913150 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:15.913232 (+ 82us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.913897 (+ 665us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.914040 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5215,"thread_start_us":91,"threads_started":1}
I20260807 02:22:15.914852 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.882137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:43016)
0807 02:22:15.882553 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.882574 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.882684 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.883040 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.883045 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.884835 (+ 1790us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:15.885187 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.885200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.886748 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.886754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.887765 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.887781 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.888633 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.888651 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.888800 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.889359 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:15.889378 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:15.907008 (+ 17630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:15.910216 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.910220 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.910222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.911723 (+ 1501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.912087 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.912089 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.912091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.912179 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.912762 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:15.912767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:15.912892 (+ 125us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.914124 (+ 1232us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.914682 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":25,"spinlock_wait_cycles":6016,"thread_start_us":131,"threads_started":1}
I20260807 02:22:15.917029 12019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.883540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.4.221.125:45039)
0807 02:22:15.883862 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.883867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.884191 (+ 324us) server_negotiation.cc:407] Connection header received
0807 02:22:15.884364 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.884367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.884421 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.884473 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.885310 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.886207 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.887470 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.887836 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.887911 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.908544 (+ 20633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.908551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.908554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.908593 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.911102 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.915560 (+ 4458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.915564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.915566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.915685 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.916015 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.916021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.916023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.916270 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.916346 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.916725 (+ 379us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.916864 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260807 02:22:15.917759 12041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.883446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45039 (local address 127.0.0.1:60366)
0807 02:22:15.884108 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.884121 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.884255 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.884614 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.884617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.884886 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:15.885159 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.885168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.886351 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.886354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.887334 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.887343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.888159 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.888164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.888256 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.888950 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:15.888966 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:15.907896 (+ 18930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:15.911436 (+ 3540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.911439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.911442 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.915419 (+ 3977us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.915805 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.915808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.915810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.915908 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.916448 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:15.916451 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:15.916541 (+ 90us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.917435 (+ 894us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.917586 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":1,"thread_start_us":105,"threads_started":1}
Aug 07 02:22:15 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069334, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.126@KRBTEST.COM for oryx/127.4.221.124@KRBTEST.COM
I20260807 02:22:15.929613 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.919682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:39679 (local address 127.0.0.1:42456)
0807 02:22:15.919884 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.919898 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.919989 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.920304 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.920307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.920524 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:15.920646 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.920652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.921645 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.921648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.922592 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.922605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.923164 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.923169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.923285 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.923872 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:15.923888 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:15.925615 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:15.927818 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.927822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.927825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.928120 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.928425 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.928432 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.928436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.928557 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.929020 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:15.929024 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:15.929142 (+ 118us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.929256 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.929429 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260807 02:22:15.929612 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.919796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.4.221.124:39679)
0807 02:22:15.919903 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.919908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.919946 (+ 38us) server_negotiation.cc:407] Connection header received
0807 02:22:15.920119 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.920123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.920178 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.920262 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.920771 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.921519 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.922726 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.923035 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.923141 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.925864 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.925869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.925871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.925899 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.927717 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.928227 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.928231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.928233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.928321 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.928667 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.928670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.928672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.928911 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.929008 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.929325 (+ 317us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.929453 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 02:22:15.931267 11263 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.221.124:39679: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c332ab71f4fb4010ac997b04a494124f"
member_type: NON_VOTER
last_known_addr {
host: "127.4.221.124"
port: 39679
}
attrs {
promote: true
}
}
dest_uuid: "9fb3edcd97e443208276838abdc871af"
cas_config_opid_index: 5
I20260807 02:22:15.931982 11263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: NON_VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: true } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } }
I20260807 02:22:15.933157 11851 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c332ab71f4fb4010ac997b04a494124f (127.4.221.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: NON_VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: true } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } } }
I20260807 02:22:15.934593 11989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:15.934917 11851 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.221.124:39679
I20260807 02:22:15.935118 11851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: NON_VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: true } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } } }
I20260807 02:22:15.935218 11851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:15.935875 11855 master.cc:537] Master@127.4.221.124:39679 shutting down...
I20260807 02:22:15.936113 11989 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 1 FOLLOWER]: Refusing update from remote peer 9fb3edcd97e443208276838abdc871af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 02:22:15.937321 11852 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 02:22:15.939436 11250 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af -> Peer c332ab71f4fb4010ac997b04a494124f (127.4.221.124:39679): Couldn't send request to peer c332ab71f4fb4010ac997b04a494124f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260807 02:22:15.942945 11855 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:15.943300 11855 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:15.943419 11855 tablet_replica.cc:330] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: stopping tablet replica
I20260807 02:22:15.946307 12019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.933477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.4.221.125:45039)
0807 02:22:15.937010 (+ 3533us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.937015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.937737 (+ 722us) server_negotiation.cc:407] Connection header received
0807 02:22:15.937991 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.937994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.938062 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.938151 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:15.941638 (+ 3487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.942527 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.944537 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.945294 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.945388 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.945585 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.946058 (+ 473us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.946158 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3426}
I20260807 02:22:15.946995 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.933223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45039 (local address 127.0.0.1:60368)
0807 02:22:15.937103 (+ 3880us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.937118 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.937853 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.938323 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.938326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.938344 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:15.941382 (+ 3038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.941393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.942817 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.942820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.944366 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.944379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.945706 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.945721 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.945864 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.945923 (+ 59us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.946736 (+ 813us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.946842 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3807}
I20260807 02:22:15.947959 11903 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 LEARNER]: Refusing update from remote peer 9fb3edcd97e443208276838abdc871af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260807 02:22:15.948552 11852 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: NON_VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260807 02:22:15.949138 11855 master.cc:552] Master@127.4.221.124:39679 shutdown complete.
I20260807 02:22:15.949191 11855 master_runner.cc:305] Clearing existing system tablet
I20260807 02:22:15.949515 11855 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:15.949584 11855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:15.951171 11855 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260807 02:22:15.951035 11903 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c332ab71f4fb4010ac997b04a494124f (127.4.221.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: NON_VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: true } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } } }
I20260807 02:22:15.952471 12045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:15.952937 11855 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:15.953074 11855 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260807 02:22:15.953109 11855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/wal
uuid: "c332ab71f4fb4010ac997b04a494124f"
format_stamp: "Formatted at 2026-08-07 02:22:15 on dist-test-slave-69rg"
server_key: "20cd91862903e502764996353df05ad0"
server_key_iv: "65e32eae9a1f14e6b0c5dbaf29b6de3a"
server_key_version: "encryptionkey@0"
I20260807 02:22:15.954183 11934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fb3edcd97e443208276838abdc871af. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: NON_VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: true } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } } }
I20260807 02:22:15.954309 11934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: This master's current role is: LEARNER
I20260807 02:22:15.954404 11934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: NON_VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: true } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } } }
I20260807 02:22:15.954466 11934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: This master's current role is: LEARNER
I20260807 02:22:15.955377 11855 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:15.957904 11855 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260807 02:22:15.958006 11855 log.cc:1196] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260807 02:22:15.958292 11855 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: Deleting consensus metadata
I20260807 02:22:15.958638 11855 master_runner.cc:315] Copying system tablet from 127.4.221.126:36267
I20260807 02:22:15.959687 11852 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af: attempting to promote NON_VOTER 54dffd44b40f425fb6309c0fa9618566 to VOTER
I20260807 02:22:15.959915 11855 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.221.126:36267
I20260807 02:22:15.960227 11852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } }
I20260807 02:22:15.962039 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.961496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:39679 (local address 127.0.0.1:42472)
0807 02:22:15.961669 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.961935 (+ 266us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 02:22:15.962705 11903 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 LEARNER]: Refusing update from remote peer 9fb3edcd97e443208276838abdc871af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260807 02:22:15.963533 11851 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260807 02:22:15.963932 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.963636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:39679 (local address 127.0.0.1:42474)
0807 02:22:15.963799 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.963854 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:39679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260807 02:22:15.964161 11250 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af -> Peer c332ab71f4fb4010ac997b04a494124f (127.4.221.124:39679): Couldn't send request to peer c332ab71f4fb4010ac997b04a494124f. Status: Network error: Client connection negotiation failed: client connection to 127.4.221.124:39679: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 02:22:15.966274 11851 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 54dffd44b40f425fb6309c0fa9618566 (127.4.221.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } } }
I20260807 02:22:15.968111 11851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } } }
I20260807 02:22:15.968235 11851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:15.968564 11851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } } }
I20260807 02:22:15.968669 11851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:15.969367 11903 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 54dffd44b40f425fb6309c0fa9618566 (127.4.221.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } } }
I20260807 02:22:15.971148 12046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } } }
I20260807 02:22:15.971285 12046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:22:15.977497 12052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.960658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:43018)
0807 02:22:15.964612 (+ 3954us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:15.964629 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:15.964797 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:15.965530 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:15.965533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:15.965790 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:15.965962 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.965970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.968874 (+ 2904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.968879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.970356 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:15.970369 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.971772 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.971775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:15.971886 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.972526 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:15.972542 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:15.973033 (+ 491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:15.975453 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.975457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.975459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.975787 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.976130 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:15.976133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:15.976135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:15.976335 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:15.976855 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:15.976858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:15.976988 (+ 130us) client_negotiation.cc:769] Sending connection context
0807 02:22:15.977102 (+ 114us) client_negotiation.cc:240] Negotiation successful
0807 02:22:15.977282 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3866,"thread_start_us":154,"threads_started":1}
I20260807 02:22:15.977510 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:15.962021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43018 (local address 127.4.221.126:36267)
0807 02:22:15.962229 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:15.962234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:15.964702 (+ 2468us) server_negotiation.cc:407] Connection header received
0807 02:22:15.965256 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:15.965259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:15.965312 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:15.965365 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:15.966089 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.966937 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.970513 (+ 3576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:15.970871 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:15.970948 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:15.973211 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:15.973215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:15.973219 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:15.973251 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:15.975310 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.975921 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.975925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.975927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.976031 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:15.976467 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:15.976470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:15.976472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:15.976738 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:15.976840 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:15.977206 (+ 366us) server_negotiation.cc:299] Negotiation successful
0807 02:22:15.977343 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260807 02:22:15.978614 11285 tablet_copy_service.cc:140] P 9fb3edcd97e443208276838abdc871af: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c332ab71f4fb4010ac997b04a494124f ({username='oryx', principal='oryx/127.4.221.124@KRBTEST.COM'} at 127.0.0.1:43018)
I20260807 02:22:15.978740 11285 tablet_copy_service.cc:161] P 9fb3edcd97e443208276838abdc871af: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c332ab71f4fb4010ac997b04a494124f at {username='oryx', principal='oryx/127.4.221.124@KRBTEST.COM'} at 127.0.0.1:43018: session id = c332ab71f4fb4010ac997b04a494124f-00000000000000000000000000000000
I20260807 02:22:15.980053 11285 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 02:22:15.981699 11855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260807 02:22:15.984477 11855 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: tablet copy: Starting download of 0 data blocks...
I20260807 02:22:15.984700 11855 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: tablet copy: Starting download of 1 WAL segments...
I20260807 02:22:15.986989 11855 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 02:22:15.989526 11285 tablet_copy_service.cc:342] P 9fb3edcd97e443208276838abdc871af: Request end of tablet copy session c332ab71f4fb4010ac997b04a494124f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.221.124@KRBTEST.COM'} at 127.0.0.1:43018
I20260807 02:22:15.989636 11285 tablet_copy_service.cc:433] P 9fb3edcd97e443208276838abdc871af: ending tablet copy session c332ab71f4fb4010ac997b04a494124f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c332ab71f4fb4010ac997b04a494124f
I20260807 02:22:15.998638 11855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:16.000748 12058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:16.000973 12055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:16.001011 12056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:16.001754 11855 server_base.cc:1034] running on GCE node
I20260807 02:22:16.002007 11855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:16.002336 11855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:16.003481 11855 hybrid_clock.cc:648] HybridClock initialized: now 1786069336003467 us; error 34 us; skew 500 ppm
Aug 07 02:22:15 dist-test-slave-69rg krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069335, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:16.005280 11855 init.cc:377] Logged in from keytab as oryx/127.4.221.124@KRBTEST.COM (short username oryx)
I20260807 02:22:16.005873 11855 webserver.cc:460] Webserver started at http://127.4.221.124:35767/ using document root <none> and password file <none>
I20260807 02:22:16.006102 11855 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:16.006287 11855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:16.008332 11855 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260807 02:22:16.010326 12066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:16.011008 11855 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260807 02:22:16.011101 11855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/wal
uuid: "c332ab71f4fb4010ac997b04a494124f"
format_stamp: "Formatted at 2026-08-07 02:22:15 on dist-test-slave-69rg"
server_key: "20cd91862903e502764996353df05ad0"
server_key_iv: "65e32eae9a1f14e6b0c5dbaf29b6de3a"
server_key_version: "encryptionkey@0"
I20260807 02:22:16.011303 11855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:16.057466 11855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:16.060240 11855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:16.065644 11855 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.124:39679
I20260807 02:22:16.066148 12125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.124:39679 every 8 connection(s)
I20260807 02:22:16.066344 11855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/master-2/data/info.pb
W20260807 02:22:16.066795 12042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:16.067166 12042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:16.067255 12042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 02:22:16.068835 12126 sys_catalog.cc:263] Verifying existing consensus state
I20260807 02:22:16.070389 12126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: Bootstrap starting.
W20260807 02:22:16.075099 12042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:16.075218 12042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:16.075341 12042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:22:16.082274 12042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.65.keytab
--principal=oryx/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:33815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=36709
--tserver_master_addrs=127.4.221.126:36267,127.4.221.125:45039,127.4.221.124:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:16.082335 12126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:22:16.082708 12126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: Bootstrap complete.
I20260807 02:22:16.083320 12126 raft_consensus.cc:348] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } }
I20260807 02:22:16.083446 12126 raft_consensus.cc:729] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c332ab71f4fb4010ac997b04a494124f, State: Initialized, Role: LEARNER
I20260807 02:22:16.083561 12126 consensus_queue.cc:260] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } }
I20260807 02:22:16.083603 12042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:16.084059 12126 sys_catalog.cc:564] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:16.083990 12129 sys_catalog.cc:455] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } } }
I20260807 02:22:16.084200 12129 sys_catalog.cc:458] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [sys.catalog]: This master's current role is: LEARNER
I20260807 02:22:16.084959 12042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 02:22:16.086995 12140 catalog_manager.cc:1259] Loaded cluster ID: c81dc73836e744b8bc84b1df20e2776b
I20260807 02:22:16.087047 12140 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: loading cluster ID for follower catalog manager: success
I20260807 02:22:16.088770 12140 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: acquiring CA information for follower catalog manager: success
I20260807 02:22:16.090140 12140 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260807 02:22:16.096463 12142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:16.096465 12145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:16.096493 12143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:16.096653 12042 server_base.cc:1034] running on GCE node
I20260807 02:22:16.097232 12042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:16.098074 12042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:16.099263 12042 hybrid_clock.cc:648] HybridClock initialized: now 1786069336099229 us; error 51 us; skew 500 ppm
Aug 07 02:22:16 dist-test-slave-69rg krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:16.102525 12042 init.cc:377] Logged in from keytab as oryx/127.4.221.65@KRBTEST.COM (short username oryx)
I20260807 02:22:16.103531 12042 webserver.cc:460] Webserver started at http://127.4.221.65:36709/ using document root <none> and password file <none>
I20260807 02:22:16.104185 12042 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:16.104298 12042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:16.108960 12042 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:22:16.111444 12152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:16.113209 12042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:22:16.113370 12042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "6904574eb48242d5828b8f304bab2f7d"
format_stamp: "Formatted at 2026-08-07 02:22:14 on dist-test-slave-69rg"
server_key: "eedc965ab8ec3745607787c1ebe435ea"
server_key_iv: "1689b9e47a13d77aba96b8172ffe2cec"
server_key_version: "encryptionkey@0"
I20260807 02:22:16.114059 12042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:16.128350 12042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:16.131858 12042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:16.132081 12042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:16.132755 12042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:16.133786 12042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:16.133929 12042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:16.134019 12042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:16.134050 12042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:16.143975 12042 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:33815
I20260807 02:22:16.144012 12269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:33815 every 8 connection(s)
I20260807 02:22:16.145104 12042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:22:16.146024 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 12042
I20260807 02:22:16.146271 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 11469
I20260807 02:22:16.156634 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:40755
--local_ip_for_outbound_sockets=127.4.221.66
--tserver_master_addrs=127.4.221.126:36267
--webserver_port=37139
--webserver_interface=127.4.221.66
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.66.keytab
--principal=oryx/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.221.126:36267,127.4.221.125:45039
--tserver_master_addrs=127.4.221.126:36267,127.4.221.125:45039,127.4.221.124:39679 with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:22:16.157119 11745 heartbeater.cc:503] Master 127.4.221.126:36267 was elected leader, sending a full tablet report...
Aug 07 02:22:16 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.124@KRBTEST.COM
Aug 07 02:22:16 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
Aug 07 02:22:16 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.125@KRBTEST.COM
I20260807 02:22:16.167757 12019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.149931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:34129 (local address 127.4.221.125:45039)
0807 02:22:16.150188 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.150193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.150900 (+ 707us) server_negotiation.cc:407] Connection header received
0807 02:22:16.151673 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.151677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.151855 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.151929 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.154244 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.155077 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.160475 (+ 5398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.160850 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.160978 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.164064 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.164067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.164069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.164096 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.165804 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.166375 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.166381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.166385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.166466 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.166795 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.166798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.166800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.166999 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.167080 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.167450 (+ 370us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.167583 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260807 02:22:16.167757 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.149765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:55441 (local address 127.4.221.126:36267)
0807 02:22:16.150074 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.150078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.150143 (+ 65us) server_negotiation.cc:407] Connection header received
0807 02:22:16.151674 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.151677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.151863 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.151929 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.154232 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.155093 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.158407 (+ 3314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.158731 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.158823 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.162560 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.162564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.162567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.162601 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.164984 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.165673 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.165679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.165681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.165777 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.166333 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.166339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.166344 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.166602 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.166686 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.167455 (+ 769us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.167583 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260807 02:22:16.168594 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.147915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:49195 (local address 127.4.221.124:39679)
0807 02:22:16.149871 (+ 1956us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.149877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.150146 (+ 269us) server_negotiation.cc:407] Connection header received
0807 02:22:16.151678 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.151684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.151754 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.151823 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.154205 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.155411 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.157678 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.158076 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.158178 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.162865 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.162872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.162878 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.162919 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.166229 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.166806 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.166811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.166814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.166915 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.167271 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.167275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.167278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.167559 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.167667 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.168177 (+ 510us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.168347 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1839,"thread_start_us":130,"threads_started":1}
I20260807 02:22:16.169540 12276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.149481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45039 (local address 127.4.221.65:34129)
0807 02:22:16.150808 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:16.150825 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:16.151428 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:16.152361 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:16.152364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:16.153088 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:16.154011 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.154029 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.155775 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.155778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.160301 (+ 4523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.160315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.161009 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.161016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.161158 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.161738 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:16.161758 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:16.163893 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:16.165952 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.165955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.165957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.166265 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.166577 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.166580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.166582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.166687 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.167139 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:16.167142 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:16.167216 (+ 74us) client_negotiation.cc:769] Sending connection context
0807 02:22:16.167927 (+ 711us) client_negotiation.cc:240] Negotiation successful
0807 02:22:16.168329 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1242,"mutex_wait_us":233,"spinlock_wait_cycles":4096,"thread_start_us":102,"threads_started":1}
I20260807 02:22:16.169541 12277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.148399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:39679 (local address 127.4.221.65:49195)
0807 02:22:16.149982 (+ 1583us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:16.150043 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:16.151400 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:16.152310 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:16.152320 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:16.153067 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:16.154006 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.154030 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.155572 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.155578 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.157528 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.157541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.158400 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.158404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.158619 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.159388 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:16.159414 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:16.162675 (+ 3261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:16.166388 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.166394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.166398 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.166672 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.167046 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.167050 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.167052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.167151 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.167735 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:16.167739 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:16.167953 (+ 214us) client_negotiation.cc:769] Sending connection context
0807 02:22:16.168817 (+ 864us) client_negotiation.cc:240] Negotiation successful
0807 02:22:16.169001 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1326,"mutex_wait_us":154,"spinlock_wait_cycles":7936,"thread_start_us":161,"threads_started":1}
I20260807 02:22:16.170673 12275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.148399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.65:55441)
0807 02:22:16.150018 (+ 1619us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:16.150060 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:16.151428 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:16.152308 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:16.152320 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:16.153066 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:16.154006 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.154029 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.155544 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.155547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.158263 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.158275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.159000 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.159004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.159110 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.159653 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:16.159671 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:16.162359 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:16.165118 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.165124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.165141 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.165537 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.165952 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.165955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.165957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.166202 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.166757 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:16.166766 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:16.167140 (+ 374us) client_negotiation.cc:769] Sending connection context
0807 02:22:16.168350 (+ 1210us) client_negotiation.cc:240] Negotiation successful
0807 02:22:16.168465 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1523,"mutex_wait_us":304,"spinlock_wait_cycles":70656,"thread_start_us":201,"threads_started":1}
I20260807 02:22:16.171156 12272 heartbeater.cc:347] Connected to a master server at 127.4.221.125:45039
I20260807 02:22:16.171157 12271 heartbeater.cc:347] Connected to a master server at 127.4.221.124:39679
I20260807 02:22:16.171438 12272 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:16.171460 12271 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:16.171625 12270 heartbeater.cc:347] Connected to a master server at 127.4.221.126:36267
I20260807 02:22:16.171741 12270 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:16.172171 12270 heartbeater.cc:511] Master 127.4.221.126:36267 requested a full tablet report, sending...
I20260807 02:22:16.172160 12272 heartbeater.cc:511] Master 127.4.221.125:45039 requested a full tablet report, sending...
I20260807 02:22:16.172158 12271 heartbeater.cc:511] Master 127.4.221.124:39679 requested a full tablet report, sending...
I20260807 02:22:16.173691 11263 ts_manager.cc:194] Re-registered known tserver with Master: 6904574eb48242d5828b8f304bab2f7d (127.4.221.65:33815)
I20260807 02:22:16.174005 11893 ts_manager.cc:194] Registered new tserver with Master: 6904574eb48242d5828b8f304bab2f7d (127.4.221.65:33815)
I20260807 02:22:16.174489 12078 ts_manager.cc:194] Registered new tserver with Master: 6904574eb48242d5828b8f304bab2f7d (127.4.221.65:33815)
I20260807 02:22:16.175544 11263 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:55441
W20260807 02:22:16.287060 12278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:16.287381 12278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:16.287429 12278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:16.291815 12278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:16.291913 12278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:16.291978 12278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:22:16.298106 12278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.66.keytab
--principal=oryx/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:40755
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=37139
--tserver_master_addrs=127.4.221.126:36267,127.4.221.125:45039,127.4.221.124:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:16.299316 12278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:16.300427 12278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:16.311261 12284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:16.311374 12283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:16.311482 12286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:16.311506 12278 server_base.cc:1034] running on GCE node
I20260807 02:22:16.312062 12278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:16.312928 12278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:16.314103 12278 hybrid_clock.cc:648] HybridClock initialized: now 1786069336314083 us; error 35 us; skew 500 ppm
Aug 07 02:22:16 dist-test-slave-69rg krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:16.317085 12278 init.cc:377] Logged in from keytab as oryx/127.4.221.66@KRBTEST.COM (short username oryx)
I20260807 02:22:16.318223 12278 webserver.cc:460] Webserver started at http://127.4.221.66:37139/ using document root <none> and password file <none>
I20260807 02:22:16.318862 12278 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:16.318982 12278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:16.323271 12278 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:22:16.325721 12293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:16.327539 12278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:22:16.327754 12278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "468cf228cf7e4d65b52b154eed448fb2"
format_stamp: "Formatted at 2026-08-07 02:22:14 on dist-test-slave-69rg"
server_key: "0ea92df9e7a33a480588bd7f1d4732d3"
server_key_iv: "f817e4bf45c767978e945fa600826c2f"
server_key_version: "encryptionkey@0"
I20260807 02:22:16.328392 12278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:16.339174 12278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:16.342681 12278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:16.342916 12278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:16.343580 12278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:16.344516 12278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:16.344592 12278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:16.344666 12278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:16.344719 12278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:16.354970 12278 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:40755
I20260807 02:22:16.355005 12410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:40755 every 8 connection(s)
I20260807 02:22:16.356212 12278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:22:16.366324 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 12278
I20260807 02:22:16.366547 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 11612
Aug 07 02:22:16 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.124@KRBTEST.COM
Aug 07 02:22:16 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
Aug 07 02:22:16 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.125@KRBTEST.COM
I20260807 02:22:16.375883 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.358183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:43755 (local address 127.4.221.124:39679)
0807 02:22:16.358355 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.358360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.359533 (+ 1173us) server_negotiation.cc:407] Connection header received
0807 02:22:16.360744 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.360748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.360809 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.360868 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.362461 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.363289 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.364784 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.365052 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.365133 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.371791 (+ 6658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.371795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.371798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.371831 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.373511 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.374052 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.374056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.374058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.374145 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.374448 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.374451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.374453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.374672 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.374752 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.375550 (+ 798us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.375706 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260807 02:22:16.375901 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.358401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:39137 (local address 127.4.221.126:36267)
0807 02:22:16.358562 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.358567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.359770 (+ 1203us) server_negotiation.cc:407] Connection header received
0807 02:22:16.360762 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.360766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.360820 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.360873 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.362461 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.363289 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.364776 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.365132 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.365257 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.371381 (+ 6124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.371386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.371389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.371426 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.373232 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.373863 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.373870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.373875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.373976 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.374423 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.374428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.374430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.374681 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.374759 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.375572 (+ 813us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.375719 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 02:22:16.378559 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:37593
--local_ip_for_outbound_sockets=127.4.221.67
--tserver_master_addrs=127.4.221.126:36267
--webserver_port=42221
--webserver_interface=127.4.221.67
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.67.keytab
--principal=oryx/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.221.126:36267,127.4.221.125:45039
--tserver_master_addrs=127.4.221.126:36267,127.4.221.125:45039,127.4.221.124:39679 with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:22:16.378861 12415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.358738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:39679 (local address 127.4.221.66:43755)
0807 02:22:16.359363 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:16.359416 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:16.360522 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:16.361108 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:16.361114 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:16.361634 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:16.362306 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.362315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.363441 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.363445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.364636 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.364646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.365316 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.365320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.365410 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.366293 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:16.366314 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:16.371607 (+ 5293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:16.373652 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.373655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.373657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.373935 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.374268 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.374271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.374273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.374355 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.374823 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:16.374830 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:16.375258 (+ 428us) client_negotiation.cc:769] Sending connection context
0807 02:22:16.376082 (+ 824us) client_negotiation.cc:240] Negotiation successful
0807 02:22:16.376676 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":523,"spinlock_wait_cycles":57472,"thread_start_us":167,"threads_started":1}
I20260807 02:22:16.380182 12412 heartbeater.cc:347] Connected to a master server at 127.4.221.124:39679
I20260807 02:22:16.380496 12412 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:16.380555 12019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.358969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:55663 (local address 127.4.221.125:45039)
0807 02:22:16.359108 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.359112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.360005 (+ 893us) server_negotiation.cc:407] Connection header received
0807 02:22:16.360710 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.360713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.360775 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.360900 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.362441 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.363154 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.364641 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.364965 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.365068 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.376529 (+ 11461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.376534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.376537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.376570 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.378421 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.378980 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.378986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.378991 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.379086 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.379434 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.379440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.379445 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.379707 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.379786 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.380178 (+ 392us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.380335 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 02:22:16.380632 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.358739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.66:39137)
0807 02:22:16.359679 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:16.359692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:16.360522 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:16.361108 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:16.361114 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:16.361632 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:16.362305 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.362315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.363442 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.363445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.364636 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.364647 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.365260 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.365264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.365378 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.366289 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:16.366339 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:16.371157 (+ 4818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:16.373379 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.373386 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.373405 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.373717 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.374089 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.374095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.374100 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.374311 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.374860 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:16.374865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:16.375258 (+ 393us) client_negotiation.cc:769] Sending connection context
0807 02:22:16.376143 (+ 885us) client_negotiation.cc:240] Negotiation successful
0807 02:22:16.376571 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":302,"spinlock_wait_cycles":35328,"thread_start_us":168,"threads_started":1}
I20260807 02:22:16.381052 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.358739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45039 (local address 127.4.221.66:55663)
0807 02:22:16.359897 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:16.359908 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:16.360503 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:16.361104 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:16.361112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:16.361630 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:16.362293 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.362306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.363303 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.363307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.364511 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.364521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.365140 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.365145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.365344 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.366283 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:16.366307 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:16.376336 (+ 10029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:16.378573 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.378579 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.378583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.378851 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.379213 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.379218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.379223 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.379328 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.379864 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:16.379870 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:16.379984 (+ 114us) client_negotiation.cc:769] Sending connection context
0807 02:22:16.380706 (+ 722us) client_negotiation.cc:240] Negotiation successful
0807 02:22:16.380885 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":214,"spinlock_wait_cycles":75392,"thread_start_us":160,"threads_started":1}
I20260807 02:22:16.381188 12412 heartbeater.cc:511] Master 127.4.221.124:39679 requested a full tablet report, sending...
I20260807 02:22:16.381325 12411 heartbeater.cc:347] Connected to a master server at 127.4.221.126:36267
I20260807 02:22:16.381405 12411 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:16.381573 12411 heartbeater.cc:511] Master 127.4.221.126:36267 requested a full tablet report, sending...
I20260807 02:22:16.381912 12413 heartbeater.cc:347] Connected to a master server at 127.4.221.125:45039
I20260807 02:22:16.381994 12413 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:16.382198 12413 heartbeater.cc:511] Master 127.4.221.125:45039 requested a full tablet report, sending...
I20260807 02:22:16.382458 11263 ts_manager.cc:194] Re-registered known tserver with Master: 468cf228cf7e4d65b52b154eed448fb2 (127.4.221.66:40755)
I20260807 02:22:16.382660 12078 ts_manager.cc:194] Registered new tserver with Master: 468cf228cf7e4d65b52b154eed448fb2 (127.4.221.66:40755)
I20260807 02:22:16.382874 11893 ts_manager.cc:194] Registered new tserver with Master: 468cf228cf7e4d65b52b154eed448fb2 (127.4.221.66:40755)
I20260807 02:22:16.383137 11263 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:39137
I20260807 02:22:16.420984 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.416853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.4.221.124:39679)
0807 02:22:16.417036 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.417040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.417057 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:22:16.417213 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.417216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.417276 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.417413 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:16.417764 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.418461 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.419954 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.420421 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.420513 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.420632 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.420796 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.420861 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260807 02:22:16.421010 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.416757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:39679 (local address 127.0.0.1:42482)
0807 02:22:16.416960 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:16.416978 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:16.417089 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:16.417433 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:16.417437 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:16.417463 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:16.417589 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.417600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.418620 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.418623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.419819 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.419832 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.420552 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.420556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.420652 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.420680 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 02:22:16.420769 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 02:22:16.420858 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260807 02:22:16.421795 12097 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260807 02:22:16.422233 12129 sys_catalog.cc:455] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fb3edcd97e443208276838abdc871af. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } } }
I20260807 02:22:16.422402 12129 sys_catalog.cc:458] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [sys.catalog]: This master's current role is: LEARNER
W20260807 02:22:16.503635 12418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:16.503955 12418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:16.504057 12418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:16.508585 12418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:16.508661 12418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:16.508733 12418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:22:16.514696 12418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.67.keytab
--principal=oryx/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:37593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=42221
--tserver_master_addrs=127.4.221.126:36267,127.4.221.125:45039,127.4.221.124:39679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:16.515905 12418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:16.516983 12418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:16.527549 12423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:16.527576 12426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:16.527563 12424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:16.527786 12418 server_base.cc:1034] running on GCE node
I20260807 02:22:16.528352 12418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:16.529065 12418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:16.530272 12418 hybrid_clock.cc:648] HybridClock initialized: now 1786069336530257 us; error 26 us; skew 500 ppm
Aug 07 02:22:16 dist-test-slave-69rg krb5kdc[11216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:16.533281 12418 init.cc:377] Logged in from keytab as oryx/127.4.221.67@KRBTEST.COM (short username oryx)
I20260807 02:22:16.534402 12418 webserver.cc:460] Webserver started at http://127.4.221.67:42221/ using document root <none> and password file <none>
I20260807 02:22:16.534956 12418 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:16.535035 12418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:16.539166 12418 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:22:16.541253 12433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:16.542621 12418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:22:16.542786 12418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4"
format_stamp: "Formatted at 2026-08-07 02:22:15 on dist-test-slave-69rg"
server_key: "c16af7422313561d2c500cc30e3fafa8"
server_key_iv: "7233f929ce07bf70af7bb75620f12593"
server_key_version: "encryptionkey@0"
I20260807 02:22:16.543358 12418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:16.573194 12418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:16.576905 12418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:16.577129 12418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:16.577872 12418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:16.578836 12418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:16.578884 12418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:16.578951 12418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:16.578984 12418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:16.589138 12418 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:37593
I20260807 02:22:16.589157 12554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:37593 every 8 connection(s)
I20260807 02:22:16.590340 12418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:22:16.599720 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 12418
Aug 07 02:22:16 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.125@KRBTEST.COM
Aug 07 02:22:16 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.124@KRBTEST.COM
Aug 07 02:22:16 dist-test-slave-69rg krb5kdc[11216](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069336, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
I20260807 02:22:16.606769 12019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.592599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:39503 (local address 127.4.221.125:45039)
0807 02:22:16.592707 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.592712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.593494 (+ 782us) server_negotiation.cc:407] Connection header received
0807 02:22:16.594631 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.594634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.594701 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.594797 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.596313 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.597147 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.598362 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.598667 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.598774 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.602102 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.602106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.602109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.602141 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.603981 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.604691 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.604698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.604702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.604798 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.605266 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.605272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.605276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.605522 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.605615 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.606430 (+ 815us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.606560 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 02:22:16.607889 11305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.592236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:47787 (local address 127.4.221.126:36267)
0807 02:22:16.592383 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.592388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.594052 (+ 1664us) server_negotiation.cc:407] Connection header received
0807 02:22:16.595037 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.595040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.595099 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.595161 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.596325 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.596877 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.598570 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.598913 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.598995 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.604045 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.604052 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.604058 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.604089 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.605890 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.606401 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.606404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.606406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.606493 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.606802 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.606808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.606813 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.607037 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.607124 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.607622 (+ 498us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.607746 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 02:22:16.607921 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.592790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45039 (local address 127.4.221.67:39503)
0807 02:22:16.593317 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:16.593373 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:16.594416 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:16.595004 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:16.595012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:16.595597 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:16.596151 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.596157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.597297 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.597300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.598234 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.598244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.598834 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.598837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.599015 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.599916 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:16.599941 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:16.601878 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:16.604118 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.604127 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.604147 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.604525 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.604933 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.604937 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.604940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.605140 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.605708 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:16.605716 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:16.606116 (+ 400us) client_negotiation.cc:769] Sending connection context
0807 02:22:16.606920 (+ 804us) client_negotiation.cc:240] Negotiation successful
0807 02:22:16.607068 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":194,"spinlock_wait_cycles":44672,"thread_start_us":152,"threads_started":1}
I20260807 02:22:16.608230 12560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.592789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:39679 (local address 127.4.221.67:40345)
0807 02:22:16.594031 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:16.594043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:16.594431 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:16.595271 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:16.595273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:16.595592 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:16.596128 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.596150 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.597266 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.597269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.598445 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.598454 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.599476 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.599481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.599590 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.600116 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:16.600131 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:16.602600 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:16.604847 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.604851 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.604854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.605097 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.605431 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.605434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.605436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.605528 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.606066 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:16.606071 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:16.606171 (+ 100us) client_negotiation.cc:769] Sending connection context
0807 02:22:16.606339 (+ 168us) client_negotiation.cc:240] Negotiation successful
0807 02:22:16.606729 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":237,"spinlock_wait_cycles":2560,"thread_start_us":109,"threads_started":1}
I20260807 02:22:16.608440 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.592789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.67:47787)
0807 02:22:16.593933 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:16.593949 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:16.594431 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:16.595222 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:16.595224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:16.595603 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:16.596147 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.596157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.597035 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.597039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.598444 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.598455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.599021 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.599024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.599105 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.600661 (+ 1556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:16.600681 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:16.603612 (+ 2931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:16.606015 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.606018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.606020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.606282 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.606612 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:16.606615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:16.606617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:16.606699 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:16.607341 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:16.607344 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:16.607430 (+ 86us) client_negotiation.cc:769] Sending connection context
0807 02:22:16.608089 (+ 659us) client_negotiation.cc:240] Negotiation successful
0807 02:22:16.608252 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":237,"spinlock_wait_cycles":3712,"thread_start_us":123,"threads_started":1}
I20260807 02:22:16.609208 12557 heartbeater.cc:347] Connected to a master server at 127.4.221.125:45039
I20260807 02:22:16.609205 12555 heartbeater.cc:347] Connected to a master server at 127.4.221.126:36267
I20260807 02:22:16.609464 12557 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:16.609465 12555 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:16.609969 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.592500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:40345 (local address 127.4.221.124:39679)
0807 02:22:16.592640 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.592644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.594110 (+ 1466us) server_negotiation.cc:407] Connection header received
0807 02:22:16.594742 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.594769 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.594830 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.594904 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.596361 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.597149 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.599095 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.599310 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.599455 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.602787 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.602791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.602794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.602828 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.604729 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.605217 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.605220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.605222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.605316 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.605644 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.605649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.605652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.605917 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.606015 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.609656 (+ 3641us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.609804 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:22:16.610003 12557 heartbeater.cc:511] Master 127.4.221.125:45039 requested a full tablet report, sending...
I20260807 02:22:16.610596 12556 heartbeater.cc:347] Connected to a master server at 127.4.221.124:39679
I20260807 02:22:16.610700 12556 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:16.610891 12556 heartbeater.cc:511] Master 127.4.221.124:39679 requested a full tablet report, sending...
I20260807 02:22:16.611236 11893 ts_manager.cc:194] Registered new tserver with Master: 40bb7a72cf2343b6beb35d6ddd62a7c4 (127.4.221.67:37593)
I20260807 02:22:16.610004 12555 heartbeater.cc:511] Master 127.4.221.126:36267 requested a full tablet report, sending...
I20260807 02:22:16.611613 12078 ts_manager.cc:194] Registered new tserver with Master: 40bb7a72cf2343b6beb35d6ddd62a7c4 (127.4.221.67:37593)
I20260807 02:22:16.612408 11263 ts_manager.cc:194] Re-registered known tserver with Master: 40bb7a72cf2343b6beb35d6ddd62a7c4 (127.4.221.67:37593)
I20260807 02:22:16.613003 11263 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:47787
I20260807 02:22:16.620062 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.602415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.4.221.126:36267)
0807 02:22:16.603165 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.603170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.608412 (+ 5242us) server_negotiation.cc:407] Connection header received
0807 02:22:16.608459 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.608462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.608521 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.608614 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.610811 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.611825 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.613360 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.613706 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.614217 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.615846 (+ 1629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.615850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.615853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.615881 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.617933 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.618500 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.618504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.618506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.618598 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.618913 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.618917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.618919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.619148 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.619298 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.619545 (+ 247us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.619685 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":104,"threads_started":1}
I20260807 02:22:16.621178 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.603172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.4.221.124:39679)
0807 02:22:16.603826 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.603831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.607268 (+ 3437us) server_negotiation.cc:407] Connection header received
0807 02:22:16.607307 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.607310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.607355 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.607431 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.611497 (+ 4066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.612216 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.614901 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.615196 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.615330 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.617136 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.617143 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.617148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.617181 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.619155 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.619758 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.619762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.619764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.619852 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.620169 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.620172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.620174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.620384 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.620525 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.620767 (+ 242us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.621022 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":112,"threads_started":1}
I20260807 02:22:16.621614 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.602751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.4.221.125:45039)
0807 02:22:16.604292 (+ 1541us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.604296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.604726 (+ 430us) server_negotiation.cc:407] Connection header received
0807 02:22:16.604771 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.604777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.604837 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.604914 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.606102 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.607705 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.615308 (+ 7603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.615644 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.615782 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.617225 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.617228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.617236 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.617260 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.619305 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.619836 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.619840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.619842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.619934 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.620293 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.620296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.620298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.620528 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.620657 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.621261 (+ 604us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.621421 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1449,"thread_start_us":94,"threads_started":1}
I20260807 02:22:16.623801 11263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:22:16.627288 11263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:22:16.630276 12054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: NON_VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: true } } }
I20260807 02:22:16.630457 12054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:16.634357 12575 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af: attempting to promote NON_VOTER c332ab71f4fb4010ac997b04a494124f to VOTER
I20260807 02:22:16.634742 12575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } }
I20260807 02:22:16.635843 12097 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 1 LEARNER]: Refusing update from remote peer 9fb3edcd97e443208276838abdc871af: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260807 02:22:16.636046 11903 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 FOLLOWER]: Refusing update from remote peer 9fb3edcd97e443208276838abdc871af: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260807 02:22:16.636407 12054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260807 02:22:16.636767 12574 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260807 02:22:16.638139 12054 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c332ab71f4fb4010ac997b04a494124f (127.4.221.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } } }
I20260807 02:22:16.638628 11903 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c332ab71f4fb4010ac997b04a494124f (127.4.221.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } } }
I20260807 02:22:16.639809 12054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } } }
I20260807 02:22:16.639950 12054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:16.639971 12574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } } }
I20260807 02:22:16.640098 12054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } } }
I20260807 02:22:16.640197 12054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:16.640098 12574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:16.640655 12097 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, c332ab71f4fb4010ac997b04a494124f (127.4.221.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } } }
I20260807 02:22:16.642109 12583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } } }
I20260807 02:22:16.642311 12583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:22:16.642549 12129 sys_catalog.cc:455] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fb3edcd97e443208276838abdc871af" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } } }
I20260807 02:22:16.642683 12129 sys_catalog.cc:458] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:22:16.655021 12587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.649484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47972 (local address 127.4.221.66:40755)
0807 02:22:16.649846 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.649861 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.649887 (+ 26us) server_negotiation.cc:407] Connection header received
0807 02:22:16.649980 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.649986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.650224 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.650404 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:16.650661 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.651934 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.653299 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.654205 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.654312 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.654488 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.654790 (+ 302us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.654890 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":122,"threads_started":1}
I20260807 02:22:16.654990 12584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.649002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:40755 (local address 127.0.0.1:47972)
0807 02:22:16.649341 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:16.649360 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:16.649463 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:16.650363 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:16.650367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:16.650388 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:16.650532 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.650543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.652076 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.652080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.653161 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.653170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.654562 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.654564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.654630 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.654648 (+ 18us) client_negotiation.cc:769] Sending connection context
0807 02:22:16.654748 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 02:22:16.654828 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":70,"thread_start_us":144,"threads_started":1}
I20260807 02:22:16.655511 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.648576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33815 (local address 127.0.0.1:37468)
0807 02:22:16.648680 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:16.648698 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:16.648776 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:16.650624 (+ 1848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:16.650627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:16.650647 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:16.650742 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.650748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.652119 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.652123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.653601 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.653613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.655057 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.655062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.655173 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.655199 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 02:22:16.655278 (+ 79us) client_negotiation.cc:240] Negotiation successful
0807 02:22:16.655352 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":54}
I20260807 02:22:16.655821 12585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.649263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:37593 (local address 127.0.0.1:58378)
0807 02:22:16.649905 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:16.649921 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:16.650003 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:16.651600 (+ 1597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:16.651602 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:16.651618 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:16.651692 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.651696 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.653111 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.653115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.654335 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.654345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.655454 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.655458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.655545 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.655569 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 02:22:16.655631 (+ 62us) client_negotiation.cc:240] Negotiation successful
0807 02:22:16.655698 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":89,"threads_started":1}
I20260807 02:22:16.655939 12588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.649812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58378 (local address 127.4.221.67:37593)
0807 02:22:16.650958 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.650974 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.650997 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:22:16.651069 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.651075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.651284 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.651438 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:16.651814 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.652985 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.654466 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.655340 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.655456 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.655604 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.655678 (+ 74us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.655776 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":109,"threads_started":1}
I20260807 02:22:16.656308 12586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.649286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37468 (local address 127.4.221.65:33815)
0807 02:22:16.650123 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.650150 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.650178 (+ 28us) server_negotiation.cc:407] Connection header received
0807 02:22:16.650241 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.650245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.650445 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.650630 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:16.650886 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.651996 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.653780 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.654718 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.654834 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.654973 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.656066 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.656190 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":90,"threads_started":1}
I20260807 02:22:16.658730 12344 tablet_service.cc:1467] Processing CreateTablet for tablet f4bb0533fd794920bc82b0f388a24e8a (DEFAULT_TABLE table=test-table [id=cea7c0e6534f4d8fb70d57a9cfa0b803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:16.658944 12202 tablet_service.cc:1467] Processing CreateTablet for tablet f4bb0533fd794920bc82b0f388a24e8a (DEFAULT_TABLE table=test-table [id=cea7c0e6534f4d8fb70d57a9cfa0b803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:16.658941 12487 tablet_service.cc:1467] Processing CreateTablet for tablet f4bb0533fd794920bc82b0f388a24e8a (DEFAULT_TABLE table=test-table [id=cea7c0e6534f4d8fb70d57a9cfa0b803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:16.659816 12344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4bb0533fd794920bc82b0f388a24e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:16.660012 12487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4bb0533fd794920bc82b0f388a24e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:16.660027 12202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4bb0533fd794920bc82b0f388a24e8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:16.667640 12589 tablet_bootstrap.cc:492] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2: Bootstrap starting.
I20260807 02:22:16.668772 12591 tablet_bootstrap.cc:492] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4: Bootstrap starting.
I20260807 02:22:16.670018 12589 tablet_bootstrap.cc:654] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:16.670212 12590 tablet_bootstrap.cc:492] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d: Bootstrap starting.
I20260807 02:22:16.670810 12591 tablet_bootstrap.cc:654] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:16.670964 12589 log.cc:824] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:16.671645 12591 log.cc:824] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:16.672986 12590 tablet_bootstrap.cc:654] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:16.673668 12589 tablet_bootstrap.cc:492] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2: No bootstrap required, opened a new log
I20260807 02:22:16.673707 12591 tablet_bootstrap.cc:492] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4: No bootstrap required, opened a new log
I20260807 02:22:16.673802 12590 log.cc:824] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:16.673933 12589 ts_tablet_manager.cc:1397] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260807 02:22:16.673986 12591 ts_tablet_manager.cc:1397] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260807 02:22:16.675956 12590 tablet_bootstrap.cc:492] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d: No bootstrap required, opened a new log
I20260807 02:22:16.676164 12590 ts_tablet_manager.cc:1397] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260807 02:22:16.678033 12591 raft_consensus.cc:348] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } }
I20260807 02:22:16.678324 12591 raft_consensus.cc:374] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:16.678388 12591 raft_consensus.cc:729] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40bb7a72cf2343b6beb35d6ddd62a7c4, State: Initialized, Role: FOLLOWER
I20260807 02:22:16.678634 12589 raft_consensus.cc:348] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } }
I20260807 02:22:16.678892 12589 raft_consensus.cc:374] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:16.678942 12589 raft_consensus.cc:729] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 468cf228cf7e4d65b52b154eed448fb2, State: Initialized, Role: FOLLOWER
I20260807 02:22:16.678895 12591 consensus_queue.cc:260] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } }
I20260807 02:22:16.679404 12589 consensus_queue.cc:260] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } }
I20260807 02:22:16.680034 12591 ts_tablet_manager.cc:1428] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 02:22:16.680194 12590 raft_consensus.cc:348] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } }
I20260807 02:22:16.680485 12555 heartbeater.cc:503] Master 127.4.221.126:36267 was elected leader, sending a full tablet report...
I20260807 02:22:16.680548 12590 raft_consensus.cc:374] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:16.680611 12590 raft_consensus.cc:729] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6904574eb48242d5828b8f304bab2f7d, State: Initialized, Role: FOLLOWER
I20260807 02:22:16.681025 12590 consensus_queue.cc:260] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } }
I20260807 02:22:16.681510 12411 heartbeater.cc:503] Master 127.4.221.126:36267 was elected leader, sending a full tablet report...
I20260807 02:22:16.681663 12270 heartbeater.cc:503] Master 127.4.221.126:36267 was elected leader, sending a full tablet report...
I20260807 02:22:16.686712 12590 ts_tablet_manager.cc:1428] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d: Time spent starting tablet: real 0.010s user 0.003s sys 0.004s
I20260807 02:22:16.686829 12589 ts_tablet_manager.cc:1428] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2: Time spent starting tablet: real 0.013s user 0.002s sys 0.005s
I20260807 02:22:16.775907 12595 raft_consensus.cc:482] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:22:16.776196 12595 raft_consensus.cc:504] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } }
I20260807 02:22:16.777808 12595 leader_election.cc:288] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6904574eb48242d5828b8f304bab2f7d (127.4.221.65:33815), 468cf228cf7e4d65b52b154eed448fb2 (127.4.221.66:40755)
I20260807 02:22:16.782388 12586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.778609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:36645 (local address 127.4.221.65:33815)
0807 02:22:16.778777 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.778782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.778799 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:22:16.778862 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.778866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.778923 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.778998 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:16.779310 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.780181 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.781286 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.781713 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.781882 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.781942 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.782186 (+ 244us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.782263 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260807 02:22:16.782388 12587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.778504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:57959 (local address 127.4.221.66:40755)
0807 02:22:16.778694 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.778701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.778719 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:22:16.778784 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.778788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.778840 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.778930 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:16.779292 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.780075 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.781275 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.781702 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.781878 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.781943 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.782169 (+ 226us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.782250 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 02:22:16.782413 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.778192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:40755 (local address 127.4.221.67:57959)
0807 02:22:16.778370 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:16.778382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:16.778433 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:16.778947 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:16.778953 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:16.778980 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:16.779148 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.779155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.780200 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.780206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.781123 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.781135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.781860 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.781864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.782017 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.782045 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 02:22:16.782142 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 02:22:16.782233 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":51}
I20260807 02:22:16.782418 12562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.778043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33815 (local address 127.4.221.67:36645)
0807 02:22:16.778297 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:16.778316 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:16.778422 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:16.779102 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:16.779106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:16.779120 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:16.779215 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.779223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.780333 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.780337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.781137 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:16.781147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.781882 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.781886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:16.782047 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.782065 (+ 18us) client_negotiation.cc:769] Sending connection context
0807 02:22:16.782169 (+ 104us) client_negotiation.cc:240] Negotiation successful
0807 02:22:16.782245 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":27}
I20260807 02:22:16.783267 12365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bb0533fd794920bc82b0f388a24e8a" candidate_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "468cf228cf7e4d65b52b154eed448fb2" is_pre_election: true
I20260807 02:22:16.783267 12223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bb0533fd794920bc82b0f388a24e8a" candidate_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6904574eb48242d5828b8f304bab2f7d" is_pre_election: true
I20260807 02:22:16.783627 12365 raft_consensus.cc:2454] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40bb7a72cf2343b6beb35d6ddd62a7c4 in term 0.
I20260807 02:22:16.783640 12223 raft_consensus.cc:2454] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40bb7a72cf2343b6beb35d6ddd62a7c4 in term 0.
I20260807 02:22:16.784185 12437 leader_election.cc:302] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 40bb7a72cf2343b6beb35d6ddd62a7c4, 468cf228cf7e4d65b52b154eed448fb2, 6904574eb48242d5828b8f304bab2f7d; no voters:
I20260807 02:22:16.784515 12595 raft_consensus.cc:2781] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:22:16.784662 12595 raft_consensus.cc:482] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:22:16.784727 12595 raft_consensus.cc:3037] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:16.786018 12595 raft_consensus.cc:504] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } }
I20260807 02:22:16.786471 12595 leader_election.cc:288] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [CANDIDATE]: Term 1 election: Requested vote from peers 6904574eb48242d5828b8f304bab2f7d (127.4.221.65:33815), 468cf228cf7e4d65b52b154eed448fb2 (127.4.221.66:40755)
I20260807 02:22:16.786898 12365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bb0533fd794920bc82b0f388a24e8a" candidate_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "468cf228cf7e4d65b52b154eed448fb2"
I20260807 02:22:16.786898 12223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4bb0533fd794920bc82b0f388a24e8a" candidate_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6904574eb48242d5828b8f304bab2f7d"
I20260807 02:22:16.787058 12223 raft_consensus.cc:3037] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:16.787058 12365 raft_consensus.cc:3037] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:16.788434 12365 raft_consensus.cc:2454] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40bb7a72cf2343b6beb35d6ddd62a7c4 in term 1.
I20260807 02:22:16.788434 12223 raft_consensus.cc:2454] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40bb7a72cf2343b6beb35d6ddd62a7c4 in term 1.
I20260807 02:22:16.788769 12435 leader_election.cc:302] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40bb7a72cf2343b6beb35d6ddd62a7c4, 468cf228cf7e4d65b52b154eed448fb2; no voters:
I20260807 02:22:16.789000 12595 raft_consensus.cc:2781] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:16.789258 12595 raft_consensus.cc:686] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 1 LEADER]: Becoming Leader. State: Replica: 40bb7a72cf2343b6beb35d6ddd62a7c4, State: Running, Role: LEADER
I20260807 02:22:16.789696 12595 consensus_queue.cc:237] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } }
I20260807 02:22:16.793780 11262 catalog_manager.cc:5495] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40bb7a72cf2343b6beb35d6ddd62a7c4 (127.4.221.67). New cstate: current_term: 1 leader_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } health_report { overall_health: HEALTHY } } }
I20260807 02:22:16.811937 12588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.807940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58388 (local address 127.4.221.67:37593)
0807 02:22:16.808106 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.808112 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.808128 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:22:16.808218 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.808220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.808278 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.808374 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:22:16.808628 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.809178 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.810139 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.810393 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.810485 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.810973 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:22:16.811505 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:22:16.811657 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.811691 (+ 34us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.811750 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:22:16.818645 12365 raft_consensus.cc:1261] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2 [term 1 FOLLOWER]: Refusing update from remote peer 40bb7a72cf2343b6beb35d6ddd62a7c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:16.818645 12223 raft_consensus.cc:1261] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d [term 1 FOLLOWER]: Refusing update from remote peer 40bb7a72cf2343b6beb35d6ddd62a7c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:16.819612 12595 consensus_queue.cc:1035] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:16.819870 12598 consensus_queue.cc:1035] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:16.830423 12601 mvcc.cc:205] Tried to move back new op lower bound from 7315740003600355328 to 7315740003493801984. Current Snapshot: MvccSnapshot[applied={T|T < 7315740003600355328}]
I20260807 02:22:16.831319 12602 mvcc.cc:205] Tried to move back new op lower bound from 7315740003600355328 to 7315740003493801984. Current Snapshot: MvccSnapshot[applied={T|T < 7315740003600355328}]
I20260807 02:22:16.833608 12603 mvcc.cc:205] Tried to move back new op lower bound from 7315740003600355328 to 7315740003493801984. Current Snapshot: MvccSnapshot[applied={T|T < 7315740003600355328}]
I20260807 02:22:16.841660 11262 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:16.841904 11262 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:16.846808 11262 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fb3edcd97e443208276838abdc871af: Sending DeleteTablet for 3 replicas of tablet f4bb0533fd794920bc82b0f388a24e8a
I20260807 02:22:16.847896 12344 tablet_service.cc:1514] Processing DeleteTablet for tablet f4bb0533fd794920bc82b0f388a24e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:16 UTC) from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:47972
I20260807 02:22:16.847893 12202 tablet_service.cc:1514] Processing DeleteTablet for tablet f4bb0533fd794920bc82b0f388a24e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:16 UTC) from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:37468
I20260807 02:22:16.848556 12613 tablet_replica.cc:330] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2: stopping tablet replica
I20260807 02:22:16.848848 12612 tablet_replica.cc:330] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d: stopping tablet replica
I20260807 02:22:16.848832 12487 tablet_service.cc:1514] Processing DeleteTablet for tablet f4bb0533fd794920bc82b0f388a24e8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:16 UTC) from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:58378
I20260807 02:22:16.849385 12612 raft_consensus.cc:2229] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:16.849601 12614 tablet_replica.cc:330] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4: stopping tablet replica
I20260807 02:22:16.849893 12614 raft_consensus.cc:2229] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:22:16.849895 12613 raft_consensus.cc:2229] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:16.850503 12614 raft_consensus.cc:2258] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:16.850966 12613 raft_consensus.cc:2258] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:16.851136 12612 raft_consensus.cc:2258] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:16.851657 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 11231
I20260807 02:22:16.855067 12612 ts_tablet_manager.cc:1905] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:16.859150 12612 ts_tablet_manager.cc:1918] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:16.859308 12612 log.cc:1196] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-0/wal/wals/f4bb0533fd794920bc82b0f388a24e8a
I20260807 02:22:16.859681 12612 ts_tablet_manager.cc:1939] T f4bb0533fd794920bc82b0f388a24e8a P 6904574eb48242d5828b8f304bab2f7d: Deleting consensus metadata
I20260807 02:22:16.861720 12613 ts_tablet_manager.cc:1905] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 02:22:16.862902 12155 connection.cc:535] server connection from 127.0.0.1:37468 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 02:22:16.866081 12613 ts_tablet_manager.cc:1918] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:16.866214 12613 log.cc:1196] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/f4bb0533fd794920bc82b0f388a24e8a
I20260807 02:22:16.866590 12613 ts_tablet_manager.cc:1939] T f4bb0533fd794920bc82b0f388a24e8a P 468cf228cf7e4d65b52b154eed448fb2: Deleting consensus metadata
I20260807 02:22:16.867269 12614 ts_tablet_manager.cc:1905] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 02:22:16.867859 12297 connection.cc:441] server connection from 127.0.0.1:47972 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47972 (request call id 1) could send its response
I20260807 02:22:16.869583 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.850256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.4.221.125:45039)
0807 02:22:16.850375 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.850380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.858322 (+ 7942us) server_negotiation.cc:407] Connection header received
0807 02:22:16.858559 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.858562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.858624 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.858691 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.860205 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.861147 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.862567 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.862975 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.863056 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.865222 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.865227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.865230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.865266 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.867370 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.868092 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.868096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.868098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.868200 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.868532 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.868536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.868537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.868788 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.868896 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.869265 (+ 369us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.869408 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260807 02:22:16.870522 12614 ts_tablet_manager.cc:1918] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:16.870659 12614 log.cc:1196] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/f4bb0533fd794920bc82b0f388a24e8a
I20260807 02:22:16.871063 12614 ts_tablet_manager.cc:1939] T f4bb0533fd794920bc82b0f388a24e8a P 40bb7a72cf2343b6beb35d6ddd62a7c4: Deleting consensus metadata
W20260807 02:22:16.872524 12435 connection.cc:441] server connection from 127.0.0.1:58378 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58378 (request call id 1) could send its response
I20260807 02:22:16.880745 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.851608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42510 (local address 127.4.221.124:39679)
0807 02:22:16.851760 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.851764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.871474 (+ 19710us) server_negotiation.cc:407] Connection header received
0807 02:22:16.871649 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.871653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.871725 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.871778 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.872728 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.873519 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.874697 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.875105 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.875221 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.876763 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.876767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.876770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.876795 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.878895 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.879375 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.879379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.879381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.879467 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.879772 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.879775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.879777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.879981 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.880081 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.880420 (+ 339us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.880555 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 02:22:16.881589 12019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.868577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.4.221.125:45039)
0807 02:22:16.868710 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.868715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.870243 (+ 1528us) server_negotiation.cc:407] Connection header received
0807 02:22:16.870532 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.870538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.870593 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.870648 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.872327 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.872995 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.873845 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.874098 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.874222 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.877403 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.877407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.877410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.877439 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.879212 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.879691 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.879694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.879697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.879790 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.880458 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.880461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.880465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.880715 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.880841 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.881244 (+ 403us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.881375 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:22:16.884852 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:16.875017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42524 (local address 127.4.221.124:39679)
0807 02:22:16.875168 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:16.875174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:16.875444 (+ 270us) server_negotiation.cc:407] Connection header received
0807 02:22:16.875552 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:16.875557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:16.875606 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:16.875693 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:16.876557 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.877312 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.878826 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:16.879199 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:16.879286 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:16.881202 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:16.881206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:16.881208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:16.881242 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:16.883029 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.883543 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.883546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.883547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.883627 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:16.883923 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:16.883925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:16.883926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:16.884144 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:16.884376 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:16.884578 (+ 202us) server_negotiation.cc:299] Negotiation successful
0807 02:22:16.884731 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 02:22:17.833595 12623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:17.832910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.65:45221)
0807 02:22:17.833346 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:17.833462 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":87,"threads_started":1}
W20260807 02:22:17.833966 12270 heartbeater.cc:644] Failed to heartbeat to 127.4.221.126:36267 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
I20260807 02:22:17.836071 12624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:17.835591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.66:44273)
0807 02:22:17.835844 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:17.835954 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":104,"threads_started":1}
W20260807 02:22:17.836442 12411 heartbeater.cc:644] Failed to heartbeat to 127.4.221.126:36267 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
I20260807 02:22:17.836532 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:17.836100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.67:42239)
0807 02:22:17.836335 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:17.836403 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
W20260807 02:22:17.836841 12555 heartbeater.cc:644] Failed to heartbeat to 127.4.221.126:36267 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
I20260807 02:22:18.347682 12626 raft_consensus.cc:482] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9fb3edcd97e443208276838abdc871af)
I20260807 02:22:18.347815 12627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9fb3edcd97e443208276838abdc871af)
I20260807 02:22:18.348112 12627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } }
I20260807 02:22:18.348129 12626 raft_consensus.cc:504] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } }
I20260807 02:22:18.349478 12626 leader_election.cc:288] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fb3edcd97e443208276838abdc871af (127.4.221.126:36267), 54dffd44b40f425fb6309c0fa9618566 (127.4.221.125:45039)
I20260807 02:22:18.350039 12627 leader_election.cc:288] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9fb3edcd97e443208276838abdc871af (127.4.221.126:36267), c332ab71f4fb4010ac997b04a494124f (127.4.221.124:39679)
I20260807 02:22:18.350404 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:18.349675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:43444)
0807 02:22:18.350036 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:18.350254 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":82,"threads_started":1}
I20260807 02:22:18.350809 12633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:18.349672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:43436)
0807 02:22:18.350598 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:18.350686 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":679,"thread_start_us":109,"threads_started":1}
W20260807 02:22:18.350756 12070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
W20260807 02:22:18.351307 11879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
I20260807 02:22:18.351938 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:18.351683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:43446)
0807 02:22:18.351809 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:18.351870 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 02:22:18.352067 12070 leader_election.cc:334] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9fb3edcd97e443208276838abdc871af (127.4.221.126:36267): Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
I20260807 02:22:18.352910 12633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:18.352610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:43454)
0807 02:22:18.352761 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:18.352837 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260807 02:22:18.353123 11879 leader_election.cc:334] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9fb3edcd97e443208276838abdc871af (127.4.221.126:36267): Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
I20260807 02:22:18.355198 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:18.350112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45039 (local address 127.0.0.1:60422)
0807 02:22:18.350588 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:18.350605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:18.350717 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:18.351314 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:18.351320 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:18.351344 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:18.351626 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:18.351635 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:18.352810 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:18.352814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:18.354050 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:18.354062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:18.354755 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:18.354759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:18.354857 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:18.354881 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 02:22:18.354958 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 02:22:18.355056 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":30,"thread_start_us":76,"threads_started":1}
I20260807 02:22:18.355233 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:18.350322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42534 (local address 127.4.221.124:39679)
0807 02:22:18.350913 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:18.350920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:18.350934 (+ 14us) server_negotiation.cc:407] Connection header received
0807 02:22:18.350985 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:18.350987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:18.351026 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:18.351136 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:18.351467 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:18.352428 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:18.354128 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:18.354567 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:18.354660 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:18.354755 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:18.355028 (+ 273us) server_negotiation.cc:299] Negotiation successful
0807 02:22:18.355113 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":528,"thread_start_us":98,"threads_started":1}
I20260807 02:22:18.355312 12632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:18.350292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.4.221.125:45039)
0807 02:22:18.350801 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:18.350806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:18.350823 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:22:18.351003 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:18.351009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:18.351082 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:18.351185 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:18.351806 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:18.352662 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:18.354192 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:18.354606 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:18.354737 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:18.354802 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:18.355000 (+ 198us) server_negotiation.cc:299] Negotiation successful
0807 02:22:18.355105 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"spinlock_wait_cycles":2304,"thread_start_us":88,"threads_started":1}
I20260807 02:22:18.355583 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:18.349672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:39679 (local address 127.0.0.1:42534)
0807 02:22:18.350091 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:18.350109 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:18.350255 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:18.351152 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:18.351155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:18.351174 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:18.351329 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:18.351337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:18.352565 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:18.352568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:18.353977 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:18.353988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:18.354696 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:18.354703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:18.354793 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:18.354814 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 02:22:18.355345 (+ 531us) client_negotiation.cc:240] Negotiation successful
0807 02:22:18.355454 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":5,"spinlock_wait_cycles":3328,"thread_start_us":118,"threads_started":1}
I20260807 02:22:18.355788 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c332ab71f4fb4010ac997b04a494124f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "54dffd44b40f425fb6309c0fa9618566" is_pre_election: true
I20260807 02:22:18.356065 11903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c332ab71f4fb4010ac997b04a494124f in term 1.
I20260807 02:22:18.356177 12097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54dffd44b40f425fb6309c0fa9618566" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c332ab71f4fb4010ac997b04a494124f" is_pre_election: true
I20260807 02:22:18.356422 12097 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54dffd44b40f425fb6309c0fa9618566 in term 1.
I20260807 02:22:18.358446 12068 leader_election.cc:302] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54dffd44b40f425fb6309c0fa9618566, c332ab71f4fb4010ac997b04a494124f; no voters: 9fb3edcd97e443208276838abdc871af
I20260807 02:22:18.358448 11880 leader_election.cc:302] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54dffd44b40f425fb6309c0fa9618566, c332ab71f4fb4010ac997b04a494124f; no voters: 9fb3edcd97e443208276838abdc871af
I20260807 02:22:18.358724 12627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 02:22:18.358814 12627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9fb3edcd97e443208276838abdc871af)
I20260807 02:22:18.358873 12627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 1 FOLLOWER]: Advancing to term 2
I20260807 02:22:18.358920 12626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 02:22:18.359023 12626 raft_consensus.cc:482] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9fb3edcd97e443208276838abdc871af)
I20260807 02:22:18.359077 12626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 1 FOLLOWER]: Advancing to term 2
I20260807 02:22:18.360164 12627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } }
I20260807 02:22:18.360420 12626 raft_consensus.cc:504] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } }
I20260807 02:22:18.360637 12627 leader_election.cc:288] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [CANDIDATE]: Term 2 election: Requested vote from peers 9fb3edcd97e443208276838abdc871af (127.4.221.126:36267), c332ab71f4fb4010ac997b04a494124f (127.4.221.124:39679)
I20260807 02:22:18.360850 12626 leader_election.cc:288] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [CANDIDATE]: Term 2 election: Requested vote from peers 9fb3edcd97e443208276838abdc871af (127.4.221.126:36267), 54dffd44b40f425fb6309c0fa9618566 (127.4.221.125:45039)
I20260807 02:22:18.360993 12097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "54dffd44b40f425fb6309c0fa9618566" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c332ab71f4fb4010ac997b04a494124f"
I20260807 02:22:18.361065 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:18.360743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:43462)
0807 02:22:18.360907 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:18.360975 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260807 02:22:18.361161 12097 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 54dffd44b40f425fb6309c0fa9618566 in current term 2: Already voted for candidate c332ab71f4fb4010ac997b04a494124f in this term.
I20260807 02:22:18.361240 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c332ab71f4fb4010ac997b04a494124f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "54dffd44b40f425fb6309c0fa9618566"
I20260807 02:22:18.361393 11903 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c332ab71f4fb4010ac997b04a494124f in current term 2: Already voted for candidate 54dffd44b40f425fb6309c0fa9618566 in this term.
I20260807 02:22:18.361394 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:18.361103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:43476)
0807 02:22:18.361265 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:18.361324 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260807 02:22:18.362131 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:18.361904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:43486)
0807 02:22:18.362027 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:18.362080 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260807 02:22:18.362154 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:18.361944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:43498)
0807 02:22:18.362055 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:18.362107 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 02:22:18.362263 12070 leader_election.cc:334] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9fb3edcd97e443208276838abdc871af (127.4.221.126:36267): Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
W20260807 02:22:18.362263 11879 leader_election.cc:334] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9fb3edcd97e443208276838abdc871af (127.4.221.126:36267): Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
I20260807 02:22:18.362337 12070 leader_election.cc:302] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c332ab71f4fb4010ac997b04a494124f; no voters: 54dffd44b40f425fb6309c0fa9618566, 9fb3edcd97e443208276838abdc871af
I20260807 02:22:18.362385 11879 leader_election.cc:302] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54dffd44b40f425fb6309c0fa9618566; no voters: 9fb3edcd97e443208276838abdc871af, c332ab71f4fb4010ac997b04a494124f
I20260807 02:22:18.362478 12626 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 02:22:18.362612 12627 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260807 02:22:18.836009 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:18.834744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.65:58465)
0807 02:22:18.835809 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:18.835886 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":951,"thread_start_us":119,"threads_started":1}
I20260807 02:22:18.837670 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:18.837148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.66:37861)
0807 02:22:18.837493 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:18.837562 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":157,"threads_started":1}
I20260807 02:22:18.838122 12638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:18.837567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.67:46243)
0807 02:22:18.837875 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:18.837971 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
I20260807 02:22:19.838202 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:19.836856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.65:60755)
0807 02:22:19.837920 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:19.838024 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":934,"thread_start_us":129,"threads_started":1}
I20260807 02:22:19.839151 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:19.838581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.66:54711)
0807 02:22:19.838928 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:19.839004 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":114,"threads_started":1}
I20260807 02:22:19.839488 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:19.838924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.67:44015)
0807 02:22:19.839171 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:19.839362 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":75,"threads_started":1}
I20260807 02:22:20.058404 12642 raft_consensus.cc:482] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:22:20.058615 12642 raft_consensus.cc:504] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } }
I20260807 02:22:20.059325 12642 leader_election.cc:288] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9fb3edcd97e443208276838abdc871af (127.4.221.126:36267), 54dffd44b40f425fb6309c0fa9618566 (127.4.221.125:45039)
I20260807 02:22:20.060196 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.059629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:51694)
0807 02:22:20.059963 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.060055 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
I20260807 02:22:20.060326 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c332ab71f4fb4010ac997b04a494124f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "54dffd44b40f425fb6309c0fa9618566" is_pre_election: true
I20260807 02:22:20.060568 11903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c332ab71f4fb4010ac997b04a494124f in term 2.
I20260807 02:22:20.061017 12068 leader_election.cc:302] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54dffd44b40f425fb6309c0fa9618566, c332ab71f4fb4010ac997b04a494124f; no voters:
I20260807 02:22:20.061308 12642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260807 02:22:20.061398 12642 raft_consensus.cc:482] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:22:20.061471 12642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 2 FOLLOWER]: Advancing to term 3
I20260807 02:22:20.061522 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.061110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:51702)
0807 02:22:20.061346 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.061431 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260807 02:22:20.061630 12070 leader_election.cc:334] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9fb3edcd97e443208276838abdc871af (127.4.221.126:36267): Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
I20260807 02:22:20.063446 12642 raft_consensus.cc:504] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } }
I20260807 02:22:20.063941 12642 leader_election.cc:288] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [CANDIDATE]: Term 3 election: Requested vote from peers 9fb3edcd97e443208276838abdc871af (127.4.221.126:36267), 54dffd44b40f425fb6309c0fa9618566 (127.4.221.125:45039)
I20260807 02:22:20.064458 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.064143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:51712)
0807 02:22:20.064274 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.064333 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 02:22:20.064834 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c332ab71f4fb4010ac997b04a494124f" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "54dffd44b40f425fb6309c0fa9618566"
I20260807 02:22:20.065110 11903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 2 FOLLOWER]: Advancing to term 3
I20260807 02:22:20.065433 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.065148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:51728)
0807 02:22:20.065306 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.065366 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260807 02:22:20.065655 12070 leader_election.cc:334] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9fb3edcd97e443208276838abdc871af (127.4.221.126:36267): Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
I20260807 02:22:20.067212 11903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c332ab71f4fb4010ac997b04a494124f in term 3.
I20260807 02:22:20.067621 12068 leader_election.cc:302] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54dffd44b40f425fb6309c0fa9618566, c332ab71f4fb4010ac997b04a494124f; no voters: 9fb3edcd97e443208276838abdc871af
I20260807 02:22:20.067873 12642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 3 FOLLOWER]: Leader election won for term 3
I20260807 02:22:20.068229 12642 raft_consensus.cc:686] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [term 3 LEADER]: Becoming Leader. State: Replica: c332ab71f4fb4010ac997b04a494124f, State: Running, Role: LEADER
I20260807 02:22:20.068593 12642 consensus_queue.cc:237] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } }
I20260807 02:22:20.070458 12645 sys_catalog.cc:455] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c332ab71f4fb4010ac997b04a494124f. Latest consensus state: current_term: 3 leader_uuid: "c332ab71f4fb4010ac997b04a494124f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } } }
I20260807 02:22:20.070626 12645 sys_catalog.cc:458] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:20.071197 12647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:20.074285 12647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:20.074753 12647 catalog_manager.cc:1259] Loaded cluster ID: c81dc73836e744b8bc84b1df20e2776b
I20260807 02:22:20.074805 12647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:20.075451 12647 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:20.075841 12647 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: Loaded TSK: 0
I20260807 02:22:20.076713 12647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:22:20.130586 12079 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:22:20.135254 12079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:22:20.139526 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.139178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:51738)
0807 02:22:20.139330 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.139413 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 02:22:20.139758 11903 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [term 3 FOLLOWER]: Refusing update from remote peer c332ab71f4fb4010ac997b04a494124f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260807 02:22:20.140434 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.140184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.0.0.1:51740)
0807 02:22:20.140293 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.140356 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260807 02:22:20.140409 12642 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [LEADER]: Connected to new peer: Peer: permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260807 02:22:20.140631 12070 consensus_peers.cc:487] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f -> Peer 9fb3edcd97e443208276838abdc871af (127.4.221.126:36267): Couldn't send request to peer 9fb3edcd97e443208276838abdc871af. Status: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260807 02:22:20.142297 12649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c332ab71f4fb4010ac997b04a494124f. Latest consensus state: current_term: 3 leader_uuid: "c332ab71f4fb4010ac997b04a494124f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } } }
I20260807 02:22:20.142462 12649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:22:20.143812 12642 sys_catalog.cc:455] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "c332ab71f4fb4010ac997b04a494124f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } } }
I20260807 02:22:20.144006 12642 sys_catalog.cc:458] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:20.144302 12649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "c332ab71f4fb4010ac997b04a494124f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "9fb3edcd97e443208276838abdc871af" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36267 } } peers { permanent_uuid: "54dffd44b40f425fb6309c0fa9618566" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45039 } attrs { promote: false } } peers { permanent_uuid: "c332ab71f4fb4010ac997b04a494124f" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 39679 } attrs { promote: false } } }
I20260807 02:22:20.144449 12649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54dffd44b40f425fb6309c0fa9618566 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:22:20.159281 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.154255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33815 (local address 127.0.0.1:47578)
0807 02:22:20.154378 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.154398 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:20.154516 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:20.155589 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:20.155592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:20.155609 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:20.155704 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:20.155712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.156889 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.156892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:20.158003 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:20.158012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.158773 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.158775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:20.158884 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:20.158912 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 02:22:20.158986 (+ 74us) client_negotiation.cc:240] Negotiation successful
0807 02:22:20.159082 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":44}
I20260807 02:22:20.159292 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.154375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47578 (local address 127.4.221.65:33815)
0807 02:22:20.155316 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:20.155322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:20.155340 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:22:20.155406 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:20.155408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:20.155464 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:20.155565 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:20.155828 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.156759 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.158150 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.158656 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.158782 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:20.158864 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:20.159024 (+ 160us) server_negotiation.cc:299] Negotiation successful
0807 02:22:20.159123 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":119,"threads_started":1}
I20260807 02:22:20.159572 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.154496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:37593 (local address 127.0.0.1:57612)
0807 02:22:20.154762 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.154776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:20.154866 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:20.155471 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:20.155474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:20.155492 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:20.155627 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:20.155641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.156918 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.156923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:20.158229 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:20.158240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.159193 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.159197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:20.159273 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:20.159303 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 02:22:20.159357 (+ 54us) client_negotiation.cc:240] Negotiation successful
0807 02:22:20.159429 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":54,"thread_start_us":129,"threads_started":1}
I20260807 02:22:20.160006 12658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.154951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:40755 (local address 127.0.0.1:40266)
0807 02:22:20.155194 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.155209 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:20.155304 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:20.156258 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:20.156261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:20.156278 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:20.156376 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:20.156382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.157386 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.157390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:20.158551 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:20.158559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.159603 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.159608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:20.159708 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:20.159728 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 02:22:20.159795 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 02:22:20.159863 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260807 02:22:20.160146 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.154848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57612 (local address 127.4.221.67:37593)
0807 02:22:20.155171 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:20.155177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:20.155194 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:22:20.155266 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:20.155272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:20.155341 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:20.155449 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:20.155903 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.156782 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.158378 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.159067 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.159757 (+ 690us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:20.159844 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:20.159884 (+ 40us) server_negotiation.cc:299] Negotiation successful
0807 02:22:20.159973 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":110,"threads_started":1}
I20260807 02:22:20.160558 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.155136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.4.221.66:40755)
0807 02:22:20.155850 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:20.155856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:20.155871 (+ 15us) server_negotiation.cc:407] Connection header received
0807 02:22:20.155939 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:20.155944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:20.156022 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:20.156121 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:20.156524 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.157254 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.158679 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.159311 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.159424 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:20.159525 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:20.160182 (+ 657us) server_negotiation.cc:299] Negotiation successful
0807 02:22:20.160376 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":130,"threads_started":1}
I20260807 02:22:20.160851 12487 tablet_service.cc:1467] Processing CreateTablet for tablet 9774e4680b1040fcb9474b29a0343591 (DEFAULT_TABLE table=test-table [id=b025a6b701864bdbbafe534d89eadcc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:20.161175 12487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9774e4680b1040fcb9474b29a0343591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:20.161150 12344 tablet_service.cc:1467] Processing CreateTablet for tablet 9774e4680b1040fcb9474b29a0343591 (DEFAULT_TABLE table=test-table [id=b025a6b701864bdbbafe534d89eadcc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:20.161420 12344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9774e4680b1040fcb9474b29a0343591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:20.162161 12202 tablet_service.cc:1467] Processing CreateTablet for tablet 9774e4680b1040fcb9474b29a0343591 (DEFAULT_TABLE table=test-table [id=b025a6b701864bdbbafe534d89eadcc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:20.162468 12202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9774e4680b1040fcb9474b29a0343591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:20.164184 12660 tablet_bootstrap.cc:492] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4: Bootstrap starting.
I20260807 02:22:20.164784 12661 tablet_bootstrap.cc:492] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2: Bootstrap starting.
I20260807 02:22:20.165509 12660 tablet_bootstrap.cc:654] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:20.165571 12662 tablet_bootstrap.cc:492] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d: Bootstrap starting.
I20260807 02:22:20.166366 12661 tablet_bootstrap.cc:654] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:20.166751 12662 tablet_bootstrap.cc:654] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:20.166967 12660 tablet_bootstrap.cc:492] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4: No bootstrap required, opened a new log
I20260807 02:22:20.167083 12660 ts_tablet_manager.cc:1397] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260807 02:22:20.167572 12660 raft_consensus.cc:348] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } }
I20260807 02:22:20.167719 12660 raft_consensus.cc:374] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:20.167766 12660 raft_consensus.cc:729] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40bb7a72cf2343b6beb35d6ddd62a7c4, State: Initialized, Role: FOLLOWER
I20260807 02:22:20.167832 12661 tablet_bootstrap.cc:492] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2: No bootstrap required, opened a new log
I20260807 02:22:20.167935 12661 ts_tablet_manager.cc:1397] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260807 02:22:20.167928 12660 consensus_queue.cc:260] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } }
I20260807 02:22:20.168212 12662 tablet_bootstrap.cc:492] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d: No bootstrap required, opened a new log
I20260807 02:22:20.168316 12662 ts_tablet_manager.cc:1397] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 02:22:20.168404 12660 ts_tablet_manager.cc:1428] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260807 02:22:20.168444 12661 raft_consensus.cc:348] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } }
I20260807 02:22:20.168561 12661 raft_consensus.cc:374] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:20.168592 12661 raft_consensus.cc:729] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 468cf228cf7e4d65b52b154eed448fb2, State: Initialized, Role: FOLLOWER
I20260807 02:22:20.168774 12661 consensus_queue.cc:260] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } }
I20260807 02:22:20.169243 12661 ts_tablet_manager.cc:1428] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260807 02:22:20.169492 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.168916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.67:58715)
0807 02:22:20.169307 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.169388 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295}
I20260807 02:22:20.170122 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.169537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.66:36603)
0807 02:22:20.169973 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.170038 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360}
I20260807 02:22:20.171147 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.170895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.66:43273)
0807 02:22:20.171032 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.171082 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 02:22:20.172269 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.170895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.67:35333)
0807 02:22:20.172120 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.172189 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1145}
I20260807 02:22:20.172787 12662 raft_consensus.cc:348] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } }
I20260807 02:22:20.172948 12662 raft_consensus.cc:374] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:20.172989 12662 raft_consensus.cc:729] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6904574eb48242d5828b8f304bab2f7d, State: Initialized, Role: FOLLOWER
I20260807 02:22:20.173151 12662 consensus_queue.cc:260] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } }
I20260807 02:22:20.173318 12412 heartbeater.cc:503] Master 127.4.221.124:39679 was elected leader, sending a full tablet report...
I20260807 02:22:20.173637 12662 ts_tablet_manager.cc:1428] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260807 02:22:20.174037 12556 heartbeater.cc:503] Master 127.4.221.124:39679 was elected leader, sending a full tablet report...
I20260807 02:22:20.174763 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.174471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.65:46583)
0807 02:22:20.174618 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.174679 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260807 02:22:20.175846 12271 heartbeater.cc:503] Master 127.4.221.124:39679 was elected leader, sending a full tablet report...
I20260807 02:22:20.176224 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.175924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.65:54959)
0807 02:22:20.176065 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.176152 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260807 02:22:20.231856 12667 raft_consensus.cc:482] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:22:20.232098 12667 raft_consensus.cc:504] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } }
I20260807 02:22:20.233783 12667 leader_election.cc:288] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6904574eb48242d5828b8f304bab2f7d (127.4.221.65:33815), 40bb7a72cf2343b6beb35d6ddd62a7c4 (127.4.221.67:37593)
I20260807 02:22:20.238602 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.234651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:35159 (local address 127.4.221.67:37593)
0807 02:22:20.234824 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:20.234829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:20.235035 (+ 206us) server_negotiation.cc:407] Connection header received
0807 02:22:20.235166 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:20.235170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:20.235240 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:20.235353 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:20.235698 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.236223 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.237386 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.237889 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.238028 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:20.238100 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:20.238342 (+ 242us) server_negotiation.cc:299] Negotiation successful
0807 02:22:20.238436 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260807 02:22:20.238610 12640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.234119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33815 (local address 127.4.221.66:36941)
0807 02:22:20.234335 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.234356 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:20.234516 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:20.235077 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:20.235083 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:20.235105 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:20.235294 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:20.235303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.236265 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.236271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:20.237242 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:20.237253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.238030 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.238034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:20.238238 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:20.238259 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 02:22:20.238355 (+ 96us) client_negotiation.cc:240] Negotiation successful
0807 02:22:20.238443 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":57}
I20260807 02:22:20.238646 12655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.234291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:36941 (local address 127.4.221.65:33815)
0807 02:22:20.234455 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:20.234460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:20.234483 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:22:20.234702 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:20.234706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:20.234780 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:20.234897 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:20.235429 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.236126 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.237411 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.237883 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.238044 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:20.238132 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:20.238397 (+ 265us) server_negotiation.cc:299] Negotiation successful
0807 02:22:20.238507 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 02:22:20.238623 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.234453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:37593 (local address 127.4.221.66:35159)
0807 02:22:20.234914 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.234931 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:20.235055 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:20.235450 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:20.235453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:20.235474 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:20.235577 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:20.235584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.236352 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.236357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:20.237247 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:20.237257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.238028 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.238032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:20.238192 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:20.238232 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 02:22:20.238324 (+ 92us) client_negotiation.cc:240] Negotiation successful
0807 02:22:20.238405 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":69,"thread_start_us":180,"threads_started":1}
I20260807 02:22:20.239351 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9774e4680b1040fcb9474b29a0343591" candidate_uuid: "468cf228cf7e4d65b52b154eed448fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" is_pre_election: true
I20260807 02:22:20.239351 12223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9774e4680b1040fcb9474b29a0343591" candidate_uuid: "468cf228cf7e4d65b52b154eed448fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6904574eb48242d5828b8f304bab2f7d" is_pre_election: true
I20260807 02:22:20.239599 12223 raft_consensus.cc:2454] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 468cf228cf7e4d65b52b154eed448fb2 in term 0.
I20260807 02:22:20.239681 12509 raft_consensus.cc:2454] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 468cf228cf7e4d65b52b154eed448fb2 in term 0.
I20260807 02:22:20.240126 12296 leader_election.cc:302] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 468cf228cf7e4d65b52b154eed448fb2, 6904574eb48242d5828b8f304bab2f7d; no voters:
I20260807 02:22:20.240455 12667 raft_consensus.cc:2781] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:22:20.240531 12667 raft_consensus.cc:482] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:22:20.240592 12667 raft_consensus.cc:3037] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:20.242120 12667 raft_consensus.cc:504] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } }
I20260807 02:22:20.242583 12667 leader_election.cc:288] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 6904574eb48242d5828b8f304bab2f7d (127.4.221.65:33815), 40bb7a72cf2343b6beb35d6ddd62a7c4 (127.4.221.67:37593)
I20260807 02:22:20.243044 12223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9774e4680b1040fcb9474b29a0343591" candidate_uuid: "468cf228cf7e4d65b52b154eed448fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6904574eb48242d5828b8f304bab2f7d"
I20260807 02:22:20.243124 12509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9774e4680b1040fcb9474b29a0343591" candidate_uuid: "468cf228cf7e4d65b52b154eed448fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4"
I20260807 02:22:20.243220 12223 raft_consensus.cc:3037] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:20.243248 12509 raft_consensus.cc:3037] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:20.244640 12509 raft_consensus.cc:2454] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 468cf228cf7e4d65b52b154eed448fb2 in term 1.
I20260807 02:22:20.244639 12223 raft_consensus.cc:2454] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 468cf228cf7e4d65b52b154eed448fb2 in term 1.
I20260807 02:22:20.244983 12296 leader_election.cc:302] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40bb7a72cf2343b6beb35d6ddd62a7c4, 468cf228cf7e4d65b52b154eed448fb2; no voters:
I20260807 02:22:20.245187 12667 raft_consensus.cc:2781] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:20.245468 12667 raft_consensus.cc:686] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [term 1 LEADER]: Becoming Leader. State: Replica: 468cf228cf7e4d65b52b154eed448fb2, State: Running, Role: LEADER
I20260807 02:22:20.245847 12667 consensus_queue.cc:237] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } }
I20260807 02:22:20.246429 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.246098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.66:43015)
0807 02:22:20.246238 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.246330 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 02:22:20.249130 12078 catalog_manager.cc:5495] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 468cf228cf7e4d65b52b154eed448fb2 (127.4.221.66). New cstate: current_term: 1 leader_uuid: "468cf228cf7e4d65b52b154eed448fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "468cf228cf7e4d65b52b154eed448fb2" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40755 } health_report { overall_health: HEALTHY } } }
I20260807 02:22:20.265100 12659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.260660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.4.221.66:40755)
0807 02:22:20.260833 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:20.260838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:20.260855 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:22:20.260985 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:20.260987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:20.261062 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:20.261167 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:22:20.261399 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.262082 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.263104 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:20.263354 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:20.263444 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:20.263819 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:22:20.264542 (+ 723us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:22:20.264648 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:20.264763 (+ 115us) server_negotiation.cc:299] Negotiation successful
0807 02:22:20.264853 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 02:22:20.269383 12223 raft_consensus.cc:1261] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d [term 1 FOLLOWER]: Refusing update from remote peer 468cf228cf7e4d65b52b154eed448fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:20.269739 12509 raft_consensus.cc:1261] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 1 FOLLOWER]: Refusing update from remote peer 468cf228cf7e4d65b52b154eed448fb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:20.270148 12670 consensus_queue.cc:1035] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6904574eb48242d5828b8f304bab2f7d" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:20.270457 12667 consensus_queue.cc:1035] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40bb7a72cf2343b6beb35d6ddd62a7c4" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:20.272526 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.272174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.67:40935)
0807 02:22:20.272344 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.272419 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260807 02:22:20.272526 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.272147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.65:34751)
0807 02:22:20.272344 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.272419 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260807 02:22:20.279345 12641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.278906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.67:34233)
0807 02:22:20.279153 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.279230 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260807 02:22:20.279482 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.279103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.65:41871)
0807 02:22:20.279327 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.279385 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260807 02:22:20.281550 12669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.281066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36267 (local address 127.4.221.66:57361)
0807 02:22:20.281366 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.281443 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:36267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222}
I20260807 02:22:20.289938 12079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:20.290213 12079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:20.295630 12079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c332ab71f4fb4010ac997b04a494124f: Sending DeleteTablet for 3 replicas of tablet 9774e4680b1040fcb9474b29a0343591
I20260807 02:22:20.296666 12487 tablet_service.cc:1514] Processing DeleteTablet for tablet 9774e4680b1040fcb9474b29a0343591 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:20 UTC) from {username='oryx', principal='oryx/127.4.221.124@KRBTEST.COM'} at 127.0.0.1:57612
I20260807 02:22:20.296725 12202 tablet_service.cc:1514] Processing DeleteTablet for tablet 9774e4680b1040fcb9474b29a0343591 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:20 UTC) from {username='oryx', principal='oryx/127.4.221.124@KRBTEST.COM'} at 127.0.0.1:47578
I20260807 02:22:20.296763 12344 tablet_service.cc:1514] Processing DeleteTablet for tablet 9774e4680b1040fcb9474b29a0343591 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:20 UTC) from {username='oryx', principal='oryx/127.4.221.124@KRBTEST.COM'} at 127.0.0.1:40266
I20260807 02:22:20.297260 12684 tablet_replica.cc:330] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4: stopping tablet replica
I20260807 02:22:20.297282 12686 tablet_replica.cc:330] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d: stopping tablet replica
I20260807 02:22:20.297441 12684 raft_consensus.cc:2229] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:20.297451 12686 raft_consensus.cc:2229] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:20.297636 12684 raft_consensus.cc:2258] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:20.297636 12686 raft_consensus.cc:2258] T 9774e4680b1040fcb9474b29a0343591 P 6904574eb48242d5828b8f304bab2f7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:20.298146 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 12042
I20260807 02:22:20.298332 12685 tablet_replica.cc:330] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2: stopping tablet replica
I20260807 02:22:20.298517 12685 raft_consensus.cc:2229] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:22:20.299471 12684 ts_tablet_manager.cc:1905] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:20.302354 12684 ts_tablet_manager.cc:1918] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:20.302446 12684 log.cc:1196] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/9774e4680b1040fcb9474b29a0343591
I20260807 02:22:20.302791 12684 ts_tablet_manager.cc:1939] T 9774e4680b1040fcb9474b29a0343591 P 40bb7a72cf2343b6beb35d6ddd62a7c4: Deleting consensus metadata
I20260807 02:22:20.303779 12070 catalog_manager.cc:4841] TS 40bb7a72cf2343b6beb35d6ddd62a7c4 (127.4.221.67:37593): tablet 9774e4680b1040fcb9474b29a0343591 (table test-table [id=b025a6b701864bdbbafe534d89eadcc2]) successfully deleted
I20260807 02:22:20.303964 12685 raft_consensus.cc:2258] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:20.304999 12685 ts_tablet_manager.cc:1905] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:20.308928 12685 ts_tablet_manager.cc:1918] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:20.309033 12685 log.cc:1196] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/9774e4680b1040fcb9474b29a0343591
I20260807 02:22:20.309376 12685 ts_tablet_manager.cc:1939] T 9774e4680b1040fcb9474b29a0343591 P 468cf228cf7e4d65b52b154eed448fb2: Deleting consensus metadata
I20260807 02:22:20.310421 12068 catalog_manager.cc:4841] TS 468cf228cf7e4d65b52b154eed448fb2 (127.4.221.66:40755): tablet 9774e4680b1040fcb9474b29a0343591 (table test-table [id=b025a6b701864bdbbafe534d89eadcc2]) successfully deleted
I20260807 02:22:20.310663 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 12278
I20260807 02:22:20.310969 12658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:20.310581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33815 (local address 127.0.0.1:47592)
0807 02:22:20.310795 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:20.310852 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:33815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260807 02:22:20.312309 12070 catalog_manager.cc:4584] TS 6904574eb48242d5828b8f304bab2f7d (127.4.221.65:33815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9774e4680b1040fcb9474b29a0343591: Network error: Client connection negotiation failed: client connection to 127.4.221.65:33815: connect: Connection refused (error 111)
I20260807 02:22:20.319401 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 12418
I20260807 02:22:20.329180 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 11750
I20260807 02:22:20.337636 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 11855
2026-08-07T02:22:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7452 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:22:20 dist-test-slave-69rg krb5kdc[12689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:22:20 dist-test-slave-69rg krb5kdc[12689](info): set up 2 sockets
Aug 07 02:22:20 dist-test-slave-69rg krb5kdc[12689](info): commencing operation
krb5kdc: starting...
W20260807 02:22:21.454391 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.003s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:22:21 dist-test-slave-69rg krb5kdc[12689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:22:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.126.keytab.
Entry for principal oryx/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.126.keytab.
I20260807 02:22:21.590250 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:43993
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.126.keytab
--principal=oryx/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:43993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:21.711395 12702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:21.711798 12702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:21.711891 12702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:21.716287 12702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:21.716418 12702 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260807 02:22:21.716449 12702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:21.716493 12702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:21.716553 12702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:21.716591 12702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:22:21.722472 12702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:43993
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.126.keytab
--principal=oryx/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:43993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:21.723774 12702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:21.724932 12702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:21.734514 12708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:21.734540 12710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:21.734536 12707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:21.734582 12702 server_base.cc:1034] running on GCE node
I20260807 02:22:21.735566 12702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:21.736900 12702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:21.738127 12702 hybrid_clock.cc:648] HybridClock initialized: now 1786069341738111 us; error 60 us; skew 500 ppm
Aug 07 02:22:21 dist-test-slave-69rg krb5kdc[12689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069341, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:21.741567 12702 init.cc:377] Logged in from keytab as oryx/127.4.221.126@KRBTEST.COM (short username oryx)
I20260807 02:22:21.742779 12702 webserver.cc:460] Webserver started at http://127.4.221.126:41471/ using document root <none> and password file <none>
I20260807 02:22:21.743415 12702 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:21.743521 12702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:21.743791 12702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:21.746169 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "29c21cb802aa4a0ba568ba4cfa1df249"
format_stamp: "Formatted at 2026-08-07 02:22:21 on dist-test-slave-69rg"
server_key: "859cc9e39e330256746a5ae0ef6255fa"
server_key_iv: "6a865f51a58a3029ebf08f1c2b8e77c0"
server_key_version: "encryptionkey@0"
I20260807 02:22:21.746810 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "29c21cb802aa4a0ba568ba4cfa1df249"
format_stamp: "Formatted at 2026-08-07 02:22:21 on dist-test-slave-69rg"
server_key: "859cc9e39e330256746a5ae0ef6255fa"
server_key_iv: "6a865f51a58a3029ebf08f1c2b8e77c0"
server_key_version: "encryptionkey@0"
I20260807 02:22:21.750931 12702 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260807 02:22:21.753511 12717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:21.754944 12702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 02:22:21.755102 12702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "29c21cb802aa4a0ba568ba4cfa1df249"
format_stamp: "Formatted at 2026-08-07 02:22:21 on dist-test-slave-69rg"
server_key: "859cc9e39e330256746a5ae0ef6255fa"
server_key_iv: "6a865f51a58a3029ebf08f1c2b8e77c0"
server_key_version: "encryptionkey@0"
I20260807 02:22:21.755230 12702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:21.771009 12702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:21.774317 12702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:21.774544 12702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:21.783310 12702 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:43993
I20260807 02:22:21.783335 12780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:43993 every 8 connection(s)
I20260807 02:22:21.784317 12702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:21.787370 12781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:21.788879 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 12702
I20260807 02:22:21.789009 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:22:21.789351 4981 external_mini_cluster.cc:1428] Setting key afb6e3c9b419287c5e4070cac5487fd0
I20260807 02:22:21.794265 12781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249: Bootstrap starting.
I20260807 02:22:21.796988 12781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:21.797971 12781 log.cc:824] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249: Log is configured to *not* fsync() on all Append() calls
Aug 07 02:22:21 dist-test-slave-69rg krb5kdc[12689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
I20260807 02:22:21.800700 12781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249: No bootstrap required, opened a new log
I20260807 02:22:21.804236 12781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c21cb802aa4a0ba568ba4cfa1df249" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43993 } }
I20260807 02:22:21.804575 12781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:21.804646 12781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c21cb802aa4a0ba568ba4cfa1df249, State: Initialized, Role: FOLLOWER
I20260807 02:22:21.805101 12781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c21cb802aa4a0ba568ba4cfa1df249" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43993 } }
I20260807 02:22:21.805234 12781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:21.805303 12781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:21.805408 12781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:21.806602 12781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c21cb802aa4a0ba568ba4cfa1df249" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43993 } }
I20260807 02:22:21.807006 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:21.791007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.4.221.126:43993)
0807 02:22:21.791474 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:21.791492 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:21.791537 (+ 45us) server_negotiation.cc:407] Connection header received
0807 02:22:21.792184 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:21.792200 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:21.792532 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:21.792818 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:21.793897 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:21.794996 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:21.796232 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:21.796690 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:21.796809 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:21.800133 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:21.800143 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:21.800165 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:21.800214 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:21.803578 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:21.804159 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:21.804164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:21.804169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:21.804294 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:21.804652 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:21.804656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:21.804658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:21.805212 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:21.805471 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:21.805791 (+ 320us) server_negotiation.cc:299] Negotiation successful
0807 02:22:21.806145 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":167,"threads_started":1}
I20260807 02:22:21.806978 12781 leader_election.cc:302] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29c21cb802aa4a0ba568ba4cfa1df249; no voters:
I20260807 02:22:21.807376 12781 leader_election.cc:288] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:22:21.807466 12786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:21.807686 12786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [term 1 LEADER]: Becoming Leader. State: Replica: 29c21cb802aa4a0ba568ba4cfa1df249, State: Running, Role: LEADER
I20260807 02:22:21.808099 12786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c21cb802aa4a0ba568ba4cfa1df249" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43993 } }
I20260807 02:22:21.808779 12781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:21.809367 12786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29c21cb802aa4a0ba568ba4cfa1df249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c21cb802aa4a0ba568ba4cfa1df249" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43993 } } }
I20260807 02:22:21.809494 12786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:21.810009 12787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29c21cb802aa4a0ba568ba4cfa1df249. Latest consensus state: current_term: 1 leader_uuid: "29c21cb802aa4a0ba568ba4cfa1df249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c21cb802aa4a0ba568ba4cfa1df249" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43993 } } }
I20260807 02:22:21.810132 12787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:21.810397 12792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:21.813614 12792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:21.819628 12792 catalog_manager.cc:1347] Generated new cluster ID: 1df6e2c35c704699ba9c0018ec4b35fa
I20260807 02:22:21.819731 12792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:21.828644 12792 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:22:21.829789 12792 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:21.837138 12792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249: Generated new TSK 0
I20260807 02:22:21.838013 12792 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.65.keytab.
Entry for principal oryx/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.65.keytab.
I20260807 02:22:21.880821 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:43993
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.65.keytab
--principal=oryx/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:22.002879 12808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:22.003206 12808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:22.003254 12808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:22.007382 12808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:22.007452 12808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:22.007493 12808 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:22:22.007527 12808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:22:22.013691 12808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.65.keytab
--principal=oryx/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.221.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:22.014829 12808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:22.015825 12808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:22.026104 12816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:22.026098 12813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:22.026098 12814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:22.026499 12808 server_base.cc:1034] running on GCE node
I20260807 02:22:22.026991 12808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:22.027695 12808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:22.028863 12808 hybrid_clock.cc:648] HybridClock initialized: now 1786069342028835 us; error 47 us; skew 500 ppm
Aug 07 02:22:22 dist-test-slave-69rg krb5kdc[12689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:22.031821 12808 init.cc:377] Logged in from keytab as oryx/127.4.221.65@KRBTEST.COM (short username oryx)
I20260807 02:22:22.032815 12808 webserver.cc:460] Webserver started at http://127.4.221.65:39543/ using document root <none> and password file <none>
I20260807 02:22:22.033452 12808 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:22.033551 12808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:22.033802 12808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:22.036068 12808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "7409551e595b4e609349f7360221932c"
format_stamp: "Formatted at 2026-08-07 02:22:22 on dist-test-slave-69rg"
server_key: "d1006e0688d2f22a7ac3c29c75ab92a6"
server_key_iv: "8680540007c195de3e0d9bee67d66183"
server_key_version: "encryptionkey@0"
I20260807 02:22:22.036696 12808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "7409551e595b4e609349f7360221932c"
format_stamp: "Formatted at 2026-08-07 02:22:22 on dist-test-slave-69rg"
server_key: "d1006e0688d2f22a7ac3c29c75ab92a6"
server_key_iv: "8680540007c195de3e0d9bee67d66183"
server_key_version: "encryptionkey@0"
I20260807 02:22:22.041016 12808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260807 02:22:22.043646 12823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:22.045042 12808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:22:22.045219 12808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "7409551e595b4e609349f7360221932c"
format_stamp: "Formatted at 2026-08-07 02:22:22 on dist-test-slave-69rg"
server_key: "d1006e0688d2f22a7ac3c29c75ab92a6"
server_key_iv: "8680540007c195de3e0d9bee67d66183"
server_key_version: "encryptionkey@0"
I20260807 02:22:22.045349 12808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:22.068868 12808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:22.072423 12808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:22.072674 12808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:22.074416 12808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:22.074494 12808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:22.074568 12808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:22.074615 12808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:22:22 dist-test-slave-69rg krb5kdc[12689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
I20260807 02:22:22.086683 12808 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:33021
I20260807 02:22:22.086820 12942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:33021 every 8 connection(s)
I20260807 02:22:22.087776 12808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:22:22.089278 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 12808
I20260807 02:22:22.089428 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:22:22.089859 4981 external_mini_cluster.cc:1428] Setting key fb2a442ca2f8d80050e9e8b65f81b88c
I20260807 02:22:22.092015 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.075317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:41719 (local address 127.4.221.126:43993)
0807 02:22:22.075512 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.075519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.076468 (+ 949us) server_negotiation.cc:407] Connection header received
0807 02:22:22.077933 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.077941 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.078022 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.078092 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:22.080338 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.080968 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.083168 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.083450 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.083556 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.087256 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:22.087260 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:22.087262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:22.087292 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:22.089060 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.089757 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.089760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.089761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.089897 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.090377 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.090380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.090382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.090764 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:22.090893 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.091700 (+ 807us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.091853 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 02:22:22.092778 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.075714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:43993 (local address 127.4.221.65:41719)
0807 02:22:22.076250 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:22.076317 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:22.077580 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:22.078658 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:22.078670 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:22.079268 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:22.080115 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.080140 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.081140 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.081144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.082968 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.082989 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.083662 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.083668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.083920 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.084772 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:22.084799 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:22.086997 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:22.089205 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:22.089213 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:22.089229 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:22.089628 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:22.090007 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:22.090011 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:22.090013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:22.090209 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:22.090894 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:22.090902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:22.091199 (+ 297us) client_negotiation.cc:769] Sending connection context
0807 02:22:22.091408 (+ 209us) client_negotiation.cc:240] Negotiation successful
0807 02:22:22.091816 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
I20260807 02:22:22.094405 12943 heartbeater.cc:347] Connected to a master server at 127.4.221.126:43993
I20260807 02:22:22.094555 12943 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:22.095142 12943 heartbeater.cc:511] Master 127.4.221.126:43993 requested a full tablet report, sending...
I20260807 02:22:22.096933 12735 ts_manager.cc:194] Registered new tserver with Master: 7409551e595b4e609349f7360221932c (127.4.221.65:33021)
I20260807 02:22:22.098769 12735 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:41719
WARNING: no policy specified for oryx/127.4.221.66@KRBTEST.COM; defaulting to no policy
I20260807 02:22:22.105602 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.096889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:43575 (local address 127.4.221.126:43993)
0807 02:22:22.097012 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.097016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.097258 (+ 242us) server_negotiation.cc:407] Connection header received
0807 02:22:22.097408 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.097411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.097450 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.097536 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:22.098053 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.098593 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.099460 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.099778 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.099880 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.101371 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:22.101375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:22.101378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:22.101406 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:22.103448 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.104022 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.104025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.104026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.104121 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.104503 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.104506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.104507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.104714 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:22.104803 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.105353 (+ 550us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.105475 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 02:22:22.105567 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.096830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:43993 (local address 127.4.221.65:43575)
0807 02:22:22.097164 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:22.097175 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:22.097302 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:22.097553 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:22.097556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:22.097775 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:22.097941 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.097947 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.098728 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.098731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.099335 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.099343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.099903 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.099909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.100026 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.100683 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:22.100704 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:22.101201 (+ 497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:22.103617 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:22.103621 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:22.103623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:22.103931 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:22.104279 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:22.104282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:22.104283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:22.104373 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:22.104960 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:22.104966 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:22.105097 (+ 131us) client_negotiation.cc:769] Sending connection context
0807 02:22:22.105207 (+ 110us) client_negotiation.cc:240] Negotiation successful
0807 02:22:22.105345 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.66.keytab.
Entry for principal oryx/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.66.keytab.
I20260807 02:22:22.131132 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:43993
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.66.keytab
--principal=oryx/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:22.251382 12952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:22.251718 12952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:22.251801 12952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:22.256299 12952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:22.256423 12952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:22.256500 12952 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:22:22.256588 12952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:22:22.262454 12952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.66.keytab
--principal=oryx/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.221.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:22.263664 12952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:22.264765 12952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:22.275153 12961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:22.275318 12952 server_base.cc:1034] running on GCE node
W20260807 02:22:22.275239 12959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:22.275153 12958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:22.276077 12952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:22.276743 12952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:22.277912 12952 hybrid_clock.cc:648] HybridClock initialized: now 1786069342277892 us; error 34 us; skew 500 ppm
Aug 07 02:22:22 dist-test-slave-69rg krb5kdc[12689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:22.280884 12952 init.cc:377] Logged in from keytab as oryx/127.4.221.66@KRBTEST.COM (short username oryx)
I20260807 02:22:22.281801 12952 webserver.cc:460] Webserver started at http://127.4.221.66:39499/ using document root <none> and password file <none>
I20260807 02:22:22.282402 12952 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:22.282462 12952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:22.282634 12952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:22.284723 12952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "b323f506ba084d9c84bcdd79fa644190"
format_stamp: "Formatted at 2026-08-07 02:22:22 on dist-test-slave-69rg"
server_key: "4d78dffeb1056f6a546832415f1c2954"
server_key_iv: "42f0a74f16ff95e00de7fc16d6e2cac5"
server_key_version: "encryptionkey@0"
I20260807 02:22:22.285334 12952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "b323f506ba084d9c84bcdd79fa644190"
format_stamp: "Formatted at 2026-08-07 02:22:22 on dist-test-slave-69rg"
server_key: "4d78dffeb1056f6a546832415f1c2954"
server_key_iv: "42f0a74f16ff95e00de7fc16d6e2cac5"
server_key_version: "encryptionkey@0"
I20260807 02:22:22.289353 12952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260807 02:22:22.292104 12968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:22.293525 12952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:22:22.293649 12952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "b323f506ba084d9c84bcdd79fa644190"
format_stamp: "Formatted at 2026-08-07 02:22:22 on dist-test-slave-69rg"
server_key: "4d78dffeb1056f6a546832415f1c2954"
server_key_iv: "42f0a74f16ff95e00de7fc16d6e2cac5"
server_key_version: "encryptionkey@0"
I20260807 02:22:22.293771 12952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:22.310717 12952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:22.313812 12952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:22.314054 12952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:22.315507 12952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:22.315559 12952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:22.315623 12952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:22.315676 12952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:22:22 dist-test-slave-69rg krb5kdc[12689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
I20260807 02:22:22.326992 12952 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:32967
I20260807 02:22:22.327420 13089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:32967 every 8 connection(s)
I20260807 02:22:22.328089 12952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:22:22.329479 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 12952
I20260807 02:22:22.329627 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:22:22.329994 4981 external_mini_cluster.cc:1428] Setting key 6752f5d49b2f45407e42186b7536037e
I20260807 02:22:22.330803 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.316455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:32927 (local address 127.4.221.126:43993)
0807 02:22:22.316586 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.316589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.317678 (+ 1089us) server_negotiation.cc:407] Connection header received
0807 02:22:22.318871 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.318875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.318937 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.319049 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:22.320604 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.321518 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.323026 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.323277 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.323399 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.326402 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:22.326406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:22.326409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:22.326446 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:22.328257 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.328883 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.328887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.328889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.328987 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.329418 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.329421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.329422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.329660 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:22.329758 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.330497 (+ 739us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.330659 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 02:22:22.331753 12978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.317017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:43993 (local address 127.4.221.66:32927)
0807 02:22:22.317527 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:22.317568 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:22.318649 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:22.319249 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:22.319257 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:22.319753 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:22.320433 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.320452 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.321696 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.321700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.322900 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.322910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.323464 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.323468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.323643 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.324323 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:22.324339 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:22.326214 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:22.328373 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:22.328378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:22.328389 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:22.328751 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:22.329105 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:22.329108 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:22.329110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:22.329302 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:22.329767 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:22.329774 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:22.330133 (+ 359us) client_negotiation.cc:769] Sending connection context
0807 02:22:22.330343 (+ 210us) client_negotiation.cc:240] Negotiation successful
0807 02:22:22.330709 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":149,"threads_started":1}
I20260807 02:22:22.334889 13090 heartbeater.cc:347] Connected to a master server at 127.4.221.126:43993
I20260807 02:22:22.335079 13090 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:22.335610 13090 heartbeater.cc:511] Master 127.4.221.126:43993 requested a full tablet report, sending...
I20260807 02:22:22.336884 12734 ts_manager.cc:194] Registered new tserver with Master: b323f506ba084d9c84bcdd79fa644190 (127.4.221.66:32967)
I20260807 02:22:22.337807 12734 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:32927
WARNING: no policy specified for oryx/127.4.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.221.67@KRBTEST.COM" created.
I20260807 02:22:22.346719 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.337849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:43993 (local address 127.4.221.66:50447)
0807 02:22:22.338254 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:22.338272 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:22.338431 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:22.338875 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:22.338881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:22.339141 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:22.339352 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.339365 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.340376 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.340382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.341341 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.341355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.341850 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.341854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.341968 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.342531 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:22.342551 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:22.342990 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:22.344890 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:22.344892 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:22.344894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:22.345177 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:22.345452 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:22.345455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:22.345456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:22.345540 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:22.346066 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:22.346069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:22.346236 (+ 167us) client_negotiation.cc:769] Sending connection context
0807 02:22:22.346360 (+ 124us) client_negotiation.cc:240] Negotiation successful
0807 02:22:22.346549 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260807 02:22:22.346728 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.337521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:50447 (local address 127.4.221.126:43993)
0807 02:22:22.337690 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.337698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.338346 (+ 648us) server_negotiation.cc:407] Connection header received
0807 02:22:22.338560 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.338564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.338620 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.338687 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:22.339495 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.340225 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.341473 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.341690 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.341770 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.343192 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:22.343196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:22.343198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:22.343221 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:22.344770 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.345280 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.345282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.345283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.345361 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.345621 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.345623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.345624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.345931 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:22.346047 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.346405 (+ 358us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.346540 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.67.keytab.
Entry for principal oryx/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.67.keytab.
I20260807 02:22:22.366431 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:43993
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.67.keytab
--principal=oryx/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:22.490229 13099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:22.490607 13099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:22.490693 13099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:22.495144 13099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:22.495260 13099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:22.495337 13099 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260807 02:22:22.495426 13099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:22:22.501495 13099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.67.keytab
--principal=oryx/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.221.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:22.502846 13099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:22.503957 13099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:22.514503 13105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:22.514616 13108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:22.514504 13104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:22.514967 13099 server_base.cc:1034] running on GCE node
I20260807 02:22:22.515444 13099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:22.516199 13099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:22.517393 13099 hybrid_clock.cc:648] HybridClock initialized: now 1786069342517360 us; error 56 us; skew 500 ppm
Aug 07 02:22:22 dist-test-slave-69rg krb5kdc[12689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:22.520715 13099 init.cc:377] Logged in from keytab as oryx/127.4.221.67@KRBTEST.COM (short username oryx)
I20260807 02:22:22.521728 13099 webserver.cc:460] Webserver started at http://127.4.221.67:41985/ using document root <none> and password file <none>
I20260807 02:22:22.522383 13099 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:22.522454 13099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:22.522639 13099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:22.524938 13099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "026602797d4341d8aa58fcbbd125db2e"
format_stamp: "Formatted at 2026-08-07 02:22:22 on dist-test-slave-69rg"
server_key: "c490071f46d9987a5e811b5bdf66fd24"
server_key_iv: "ea2c7d4e26d6af40c33a090566e8896c"
server_key_version: "encryptionkey@0"
I20260807 02:22:22.525492 13099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "026602797d4341d8aa58fcbbd125db2e"
format_stamp: "Formatted at 2026-08-07 02:22:22 on dist-test-slave-69rg"
server_key: "c490071f46d9987a5e811b5bdf66fd24"
server_key_iv: "ea2c7d4e26d6af40c33a090566e8896c"
server_key_version: "encryptionkey@0"
I20260807 02:22:22.529561 13099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260807 02:22:22.532454 13115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:22.533907 13099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 02:22:22.534055 13099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "026602797d4341d8aa58fcbbd125db2e"
format_stamp: "Formatted at 2026-08-07 02:22:22 on dist-test-slave-69rg"
server_key: "c490071f46d9987a5e811b5bdf66fd24"
server_key_iv: "ea2c7d4e26d6af40c33a090566e8896c"
server_key_version: "encryptionkey@0"
I20260807 02:22:22.534222 13099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:22.552873 13099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:22.556159 13099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:22.556342 13099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:22.557746 13099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:22.557796 13099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:22.557927 13099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:22.557967 13099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 07 02:22:22 dist-test-slave-69rg krb5kdc[12689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069342, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
I20260807 02:22:22.569247 13099 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:44083
I20260807 02:22:22.569317 13234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:44083 every 8 connection(s)
I20260807 02:22:22.570268 13099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:22:22.572973 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.558720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:51939 (local address 127.4.221.126:43993)
0807 02:22:22.558881 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.558886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.559812 (+ 926us) server_negotiation.cc:407] Connection header received
0807 02:22:22.560902 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.560906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.560966 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.561061 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:22.562769 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.563434 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.565363 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.565741 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.565884 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.569041 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:22.569046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:22.569049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:22.569092 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:22.570938 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.571547 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.571550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.571551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.571636 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.571942 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.571944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.571945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.572138 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:22.572215 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.572730 (+ 515us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.572847 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:22:22.573777 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.559156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:43993 (local address 127.4.221.67:51939)
0807 02:22:22.559642 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:22.559693 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:22.560650 (+ 957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:22.561285 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:22.561298 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:22.561853 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:22.562582 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.562607 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.563571 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.563577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.565218 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.565231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.565960 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.565965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.566169 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.566858 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:22.566883 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:22.568817 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:22.571077 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:22.571082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:22.571098 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:22.571440 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:22.571727 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:22.571729 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:22.571731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:22.571857 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:22.572230 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:22.572235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:22.572479 (+ 244us) client_negotiation.cc:769] Sending connection context
0807 02:22:22.572636 (+ 157us) client_negotiation.cc:240] Negotiation successful
0807 02:22:22.572896 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":138,"threads_started":1}
I20260807 02:22:22.574991 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 13099
I20260807 02:22:22.575129 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:22:22.575235 13235 heartbeater.cc:347] Connected to a master server at 127.4.221.126:43993
I20260807 02:22:22.575404 13235 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:22.575500 4981 external_mini_cluster.cc:1428] Setting key eeba2d356cf3b25074ab3171f54cd70e
I20260807 02:22:22.575986 13235 heartbeater.cc:511] Master 127.4.221.126:43993 requested a full tablet report, sending...
I20260807 02:22:22.577164 12734 ts_manager.cc:194] Registered new tserver with Master: 026602797d4341d8aa58fcbbd125db2e (127.4.221.67:44083)
I20260807 02:22:22.577967 12734 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:51939
I20260807 02:22:22.585354 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.577839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:43993 (local address 127.4.221.67:41623)
0807 02:22:22.578144 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:22.578158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:22.578265 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:22.578569 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:22.578571 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:22.578777 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:22.578885 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.578891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.579868 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.579871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.580437 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.580445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.580927 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.580930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.581009 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.581605 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:22.581619 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:22.582049 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:22.583809 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:22.583812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:22.583813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:22.584026 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:22.584336 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:22.584338 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:22.584340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:22.584428 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:22.584846 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:22.584848 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:22.584934 (+ 86us) client_negotiation.cc:769] Sending connection context
0807 02:22:22.585017 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 02:22:22.585199 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":125,"threads_started":1}
I20260807 02:22:22.585383 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.577783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:41623 (local address 127.4.221.126:43993)
0807 02:22:22.577947 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.577952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.578228 (+ 276us) server_negotiation.cc:407] Connection header received
0807 02:22:22.578365 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.578368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.578412 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.578504 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:22.579007 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.579726 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.580560 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.580791 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.580884 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.582241 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:22.582245 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:22.582247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:22.582277 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:22.583668 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.584129 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.584131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.584133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.584225 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.584546 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.584549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.584550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.584756 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:22.584840 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.585135 (+ 295us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.585252 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260807 02:22:22.589943 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 07 02:22:22 dist-test-slave-69rg krb5kdc[12689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069342, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 07 02:22:22 dist-test-slave-69rg krb5kdc[12689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:22 dist-test-slave-69rg krb5kdc[12689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 02:22:22.616835 12784 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:22.609732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50068 (local address 127.4.221.126:43993)
0807 02:22:22.609892 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.609896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.610087 (+ 191us) server_negotiation.cc:407] Connection header received
0807 02:22:22.610263 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.610265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.610312 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.610438 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:22.611294 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.611838 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.612749 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.612973 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.613087 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.616669 (+ 3582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50068: BlockingRecv error: recv got EOF from 127.0.0.1:50068 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 07 02:22:22 dist-test-slave-69rg krb5kdc[12689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069342, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
I20260807 02:22:22.626621 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.618569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50078 (local address 127.4.221.126:43993)
0807 02:22:22.618733 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.618737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.618843 (+ 106us) server_negotiation.cc:407] Connection header received
0807 02:22:22.618938 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.618940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.618979 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.619053 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:22.619613 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.620117 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.620914 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.621114 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.621201 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.623687 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:22.623690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:22.623692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:22.623726 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:22.624996 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.625391 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.625394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.625395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.625467 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.625757 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.625759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.625760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.626004 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:22.626117 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.626384 (+ 267us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.626487 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260807 02:22:22.629763 12734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:22:22.632957 12734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:22:22.649425 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.643973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59766 (local address 127.4.221.65:33021)
0807 02:22:22.644394 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.644402 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.644453 (+ 51us) server_negotiation.cc:407] Connection header received
0807 02:22:22.644639 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.644643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.644788 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.644931 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:22.645198 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.645980 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.647997 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.648677 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.648804 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.648969 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.649112 (+ 143us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.649232 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":78,"threads_started":1}
I20260807 02:22:22.649425 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.642837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:32967 (local address 127.0.0.1:35190)
0807 02:22:22.644146 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:22.644159 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:22.644230 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:22.644949 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:22.644952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:22.644971 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:22.645070 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.645076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.646619 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.646622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.647665 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.647675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.648889 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.648892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.648990 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.649023 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 02:22:22.649075 (+ 52us) client_negotiation.cc:240] Negotiation successful
0807 02:22:22.649243 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1237,"mutex_wait_us":34,"thread_start_us":100,"threads_started":1}
I20260807 02:22:22.649724 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.643481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35190 (local address 127.4.221.66:32967)
0807 02:22:22.644501 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.644514 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.644536 (+ 22us) server_negotiation.cc:407] Connection header received
0807 02:22:22.644615 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.644620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.644794 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.645227 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:22.645368 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.646483 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.647804 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.648699 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.649237 (+ 538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.649403 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.649454 (+ 51us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.649556 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919,"thread_start_us":129,"threads_started":1}
I20260807 02:22:22.650257 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.642934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.4.221.67:44083)
0807 02:22:22.643297 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.643311 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.643827 (+ 516us) server_negotiation.cc:407] Connection header received
0807 02:22:22.644059 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.644064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.644295 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.644455 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:22.644957 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.646176 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.647938 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.648707 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.648814 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.648993 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.649991 (+ 998us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.650099 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":144,"threads_started":1}
I20260807 02:22:22.649487 13259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.642806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:44083 (local address 127.0.0.1:54928)
0807 02:22:22.643679 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:22.643736 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:22.643887 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:22.644646 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:22.644652 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:22.644675 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:22.644815 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.644822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.646298 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.646301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.647799 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.647811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.649052 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.649058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.649140 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.649162 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 02:22:22.649227 (+ 65us) client_negotiation.cc:240] Negotiation successful
0807 02:22:22.649300 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":58,"thread_start_us":95,"threads_started":1}
I20260807 02:22:22.650985 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.643552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33021 (local address 127.0.0.1:59766)
0807 02:22:22.644370 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:22.644384 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:22.644488 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:22.644923 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:22.644926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:22.644944 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:22.645051 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.645058 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.646103 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.646107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.647898 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.647908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.648806 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.648809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.648924 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.648963 (+ 39us) client_negotiation.cc:769] Sending connection context
0807 02:22:22.649083 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 02:22:22.650022 (+ 939us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"mutex_wait_us":109,"thread_start_us":95,"threads_started":1}
I20260807 02:22:22.652983 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 01f71383cefd4acfbcead068110b51c7 (DEFAULT_TABLE table=test-table [id=b597f3111c78419ba69556144f2ff38f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:22.652933 13021 tablet_service.cc:1467] Processing CreateTablet for tablet 01f71383cefd4acfbcead068110b51c7 (DEFAULT_TABLE table=test-table [id=b597f3111c78419ba69556144f2ff38f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:22.654098 13021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f71383cefd4acfbcead068110b51c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:22.654117 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f71383cefd4acfbcead068110b51c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:22.653915 12875 tablet_service.cc:1467] Processing CreateTablet for tablet 01f71383cefd4acfbcead068110b51c7 (DEFAULT_TABLE table=test-table [id=b597f3111c78419ba69556144f2ff38f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:22.654891 12875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f71383cefd4acfbcead068110b51c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:22.661213 13264 tablet_bootstrap.cc:492] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e: Bootstrap starting.
I20260807 02:22:22.662860 13266 tablet_bootstrap.cc:492] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190: Bootstrap starting.
I20260807 02:22:22.663735 13264 tablet_bootstrap.cc:654] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:22.664532 13264 log.cc:824] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:22.664865 13266 tablet_bootstrap.cc:654] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:22.665745 13266 log.cc:824] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:22.665894 13265 tablet_bootstrap.cc:492] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c: Bootstrap starting.
I20260807 02:22:22.667076 13264 tablet_bootstrap.cc:492] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e: No bootstrap required, opened a new log
I20260807 02:22:22.667295 13264 ts_tablet_manager.cc:1397] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 02:22:22.667725 13266 tablet_bootstrap.cc:492] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190: No bootstrap required, opened a new log
I20260807 02:22:22.667945 13266 ts_tablet_manager.cc:1397] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260807 02:22:22.668329 13265 tablet_bootstrap.cc:654] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:22.669229 13265 log.cc:824] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:22.671247 13266 raft_consensus.cc:348] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } }
I20260807 02:22:22.671490 13265 tablet_bootstrap.cc:492] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c: No bootstrap required, opened a new log
I20260807 02:22:22.671556 13266 raft_consensus.cc:374] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:22.671623 13266 raft_consensus.cc:729] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b323f506ba084d9c84bcdd79fa644190, State: Initialized, Role: FOLLOWER
I20260807 02:22:22.671677 13265 ts_tablet_manager.cc:1397] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 02:22:22.671687 13264 raft_consensus.cc:348] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } }
I20260807 02:22:22.671953 13264 raft_consensus.cc:374] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:22.672024 13264 raft_consensus.cc:729] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026602797d4341d8aa58fcbbd125db2e, State: Initialized, Role: FOLLOWER
I20260807 02:22:22.672120 13266 consensus_queue.cc:260] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } }
I20260807 02:22:22.672503 13264 consensus_queue.cc:260] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } }
I20260807 02:22:22.673226 13266 ts_tablet_manager.cc:1428] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260807 02:22:22.673441 13264 ts_tablet_manager.cc:1428] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 02:22:22.673645 13090 heartbeater.cc:503] Master 127.4.221.126:43993 was elected leader, sending a full tablet report...
I20260807 02:22:22.673804 13235 heartbeater.cc:503] Master 127.4.221.126:43993 was elected leader, sending a full tablet report...
I20260807 02:22:22.677420 13265 raft_consensus.cc:348] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } }
I20260807 02:22:22.677729 13265 raft_consensus.cc:374] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:22.677801 13265 raft_consensus.cc:729] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7409551e595b4e609349f7360221932c, State: Initialized, Role: FOLLOWER
I20260807 02:22:22.678351 13265 consensus_queue.cc:260] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } }
I20260807 02:22:22.679008 12943 heartbeater.cc:503] Master 127.4.221.126:43993 was elected leader, sending a full tablet report...
I20260807 02:22:22.679358 13265 ts_tablet_manager.cc:1428] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c: Time spent starting tablet: real 0.008s user 0.002s sys 0.006s
I20260807 02:22:22.700917 13272 raft_consensus.cc:482] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:22:22.701227 13272 raft_consensus.cc:504] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } }
I20260807 02:22:22.702536 13272 leader_election.cc:288] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 026602797d4341d8aa58fcbbd125db2e (127.4.221.67:44083), b323f506ba084d9c84bcdd79fa644190 (127.4.221.66:32967)
I20260807 02:22:22.707799 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.703206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:37657 (local address 127.4.221.66:32967)
0807 02:22:22.703367 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.703371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.703458 (+ 87us) server_negotiation.cc:407] Connection header received
0807 02:22:22.703703 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.703705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.703748 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.703842 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:22.704185 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.704777 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.706773 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.707190 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.707328 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.707385 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.707572 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.707644 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 02:22:22.707829 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.702992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:34609 (local address 127.4.221.67:44083)
0807 02:22:22.703157 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.703162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.703494 (+ 332us) server_negotiation.cc:407] Connection header received
0807 02:22:22.703666 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.703670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.703733 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.703795 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:22.704155 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.704836 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.706742 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.707182 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.707336 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.707395 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.707608 (+ 213us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.707691 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 02:22:22.707875 13274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.702887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:32967 (local address 127.4.221.65:37657)
0807 02:22:22.703355 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:22.703369 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:22.703546 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:22.703871 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:22.703874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:22.703891 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:22.704043 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.704049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.704892 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.704899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.706597 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.706610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.707321 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.707325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.707426 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.707450 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 02:22:22.707553 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 02:22:22.707640 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":216,"spinlock_wait_cycles":55936,"thread_start_us":174,"threads_started":1}
I20260807 02:22:22.707876 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.702887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:44083 (local address 127.4.221.65:34609)
0807 02:22:22.703371 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:22.703385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:22.703547 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:22.703835 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:22.703838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:22.703855 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:22.704025 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.704032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.704941 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.704944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.706597 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.706610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.707336 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.707340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.707470 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.707498 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 02:22:22.707591 (+ 93us) client_negotiation.cc:240] Negotiation successful
0807 02:22:22.707677 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":291,"spinlock_wait_cycles":18176,"thread_start_us":95,"threads_started":1}
I20260807 02:22:22.708621 13042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f71383cefd4acfbcead068110b51c7" candidate_uuid: "7409551e595b4e609349f7360221932c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b323f506ba084d9c84bcdd79fa644190" is_pre_election: true
I20260807 02:22:22.708662 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f71383cefd4acfbcead068110b51c7" candidate_uuid: "7409551e595b4e609349f7360221932c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026602797d4341d8aa58fcbbd125db2e" is_pre_election: true
I20260807 02:22:22.708987 13042 raft_consensus.cc:2454] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7409551e595b4e609349f7360221932c in term 0.
I20260807 02:22:22.708992 13187 raft_consensus.cc:2454] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7409551e595b4e609349f7360221932c in term 0.
I20260807 02:22:22.709527 12826 leader_election.cc:302] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026602797d4341d8aa58fcbbd125db2e, 7409551e595b4e609349f7360221932c; no voters:
I20260807 02:22:22.709872 13272 raft_consensus.cc:2781] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:22:22.709986 13272 raft_consensus.cc:482] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:22:22.710047 13272 raft_consensus.cc:3037] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:22.711385 13272 raft_consensus.cc:504] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } }
I20260807 02:22:22.711834 13272 leader_election.cc:288] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [CANDIDATE]: Term 1 election: Requested vote from peers 026602797d4341d8aa58fcbbd125db2e (127.4.221.67:44083), b323f506ba084d9c84bcdd79fa644190 (127.4.221.66:32967)
I20260807 02:22:22.712262 13042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f71383cefd4acfbcead068110b51c7" candidate_uuid: "7409551e595b4e609349f7360221932c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b323f506ba084d9c84bcdd79fa644190"
I20260807 02:22:22.712363 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f71383cefd4acfbcead068110b51c7" candidate_uuid: "7409551e595b4e609349f7360221932c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026602797d4341d8aa58fcbbd125db2e"
I20260807 02:22:22.712437 13042 raft_consensus.cc:3037] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:22.712529 13187 raft_consensus.cc:3037] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:22.713606 13042 raft_consensus.cc:2454] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7409551e595b4e609349f7360221932c in term 1.
I20260807 02:22:22.713657 13187 raft_consensus.cc:2454] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7409551e595b4e609349f7360221932c in term 1.
I20260807 02:22:22.714013 12827 leader_election.cc:302] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 026602797d4341d8aa58fcbbd125db2e, 7409551e595b4e609349f7360221932c, b323f506ba084d9c84bcdd79fa644190; no voters:
I20260807 02:22:22.714250 13272 raft_consensus.cc:2781] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:22.714524 13272 raft_consensus.cc:686] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [term 1 LEADER]: Becoming Leader. State: Replica: 7409551e595b4e609349f7360221932c, State: Running, Role: LEADER
I20260807 02:22:22.714887 13272 consensus_queue.cc:237] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } }
I20260807 02:22:22.718389 12734 catalog_manager.cc:5495] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7409551e595b4e609349f7360221932c (127.4.221.65). New cstate: current_term: 1 leader_uuid: "7409551e595b4e609349f7360221932c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } health_report { overall_health: HEALTHY } } }
I20260807 02:22:22.729993 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.726162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59768 (local address 127.4.221.65:33021)
0807 02:22:22.726319 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.726324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.726341 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:22:22.726486 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.726488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.726548 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.726672 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:22:22.726960 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.727497 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.728402 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.728649 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.728763 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.729076 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:22:22.729547 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:22:22.729687 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.729751 (+ 64us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.729835 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 02:22:22.736932 13042 raft_consensus.cc:1261] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Refusing update from remote peer 7409551e595b4e609349f7360221932c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:22.736932 13187 raft_consensus.cc:1261] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Refusing update from remote peer 7409551e595b4e609349f7360221932c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:22.737684 13272 consensus_queue.cc:1035] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [LEADER]: Connected to new peer: Peer: permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:22.737988 13275 consensus_queue.cc:1035] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:22.745672 13278 mvcc.cc:205] Tried to move back new op lower bound from 7315740027840266240 to 7315740027763150848. Current Snapshot: MvccSnapshot[applied={T|T < 7315740027840266240}]
I20260807 02:22:22.746075 13279 mvcc.cc:205] Tried to move back new op lower bound from 7315740027840266240 to 7315740027763150848. Current Snapshot: MvccSnapshot[applied={T|T < 7315740027840266240}]
I20260807 02:22:22.756299 12731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:22.756479 12731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:22.759377 12731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249: Sending DeleteTablet for 3 replicas of tablet 01f71383cefd4acfbcead068110b51c7
I20260807 02:22:22.760363 13021 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f71383cefd4acfbcead068110b51c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:22 UTC) from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:35190
I20260807 02:22:22.760367 13166 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f71383cefd4acfbcead068110b51c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:22 UTC) from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:54928
I20260807 02:22:22.760978 13289 tablet_replica.cc:330] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e: stopping tablet replica
I20260807 02:22:22.761216 12875 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f71383cefd4acfbcead068110b51c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:22 UTC) from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:59766
I20260807 02:22:22.761549 13289 raft_consensus.cc:2229] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:22.761703 13290 tablet_replica.cc:330] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c: stopping tablet replica
I20260807 02:22:22.762008 13290 raft_consensus.cc:2229] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:22:22.761910 12731 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 02:22:22.762055 13289 raft_consensus.cc:2258] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 02:22:22.762377 12731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:22:22.762444 13290 raft_consensus.cc:2258] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:22.764061 13289 ts_tablet_manager.cc:1905] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:22.764061 13290 ts_tablet_manager.cc:1905] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:22.766975 13289 ts_tablet_manager.cc:1918] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:22.767175 13289 log.cc:1196] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/01f71383cefd4acfbcead068110b51c7
I20260807 02:22:22.767596 13289 ts_tablet_manager.cc:1939] T 01f71383cefd4acfbcead068110b51c7 P 026602797d4341d8aa58fcbbd125db2e: Deleting consensus metadata
I20260807 02:22:22.767783 13288 tablet_replica.cc:330] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190: stopping tablet replica
I20260807 02:22:22.768013 13021 tablet_service.cc:1467] Processing CreateTablet for tablet 39cff190bc37421cb2e189947dda9209 (DEFAULT_TABLE table=test-table [id=5cab28bb907742b6bee3c5f5d228b2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:22.768323 13021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39cff190bc37421cb2e189947dda9209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:22.768420 13290 ts_tablet_manager.cc:1918] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:22.768530 13290 log.cc:1196] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/wal/wals/01f71383cefd4acfbcead068110b51c7
I20260807 02:22:22.768869 13290 ts_tablet_manager.cc:1939] T 01f71383cefd4acfbcead068110b51c7 P 7409551e595b4e609349f7360221932c: Deleting consensus metadata
I20260807 02:22:22.769026 13288 raft_consensus.cc:2229] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:22.769430 13288 raft_consensus.cc:2258] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:22.770797 13266 tablet_bootstrap.cc:492] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190: Bootstrap starting.
I20260807 02:22:22.771325 12875 tablet_service.cc:1467] Processing CreateTablet for tablet 39cff190bc37421cb2e189947dda9209 (DEFAULT_TABLE table=test-table [id=5cab28bb907742b6bee3c5f5d228b2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:22.771569 12875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39cff190bc37421cb2e189947dda9209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:22.771842 13266 tablet_bootstrap.cc:654] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:22.772185 13288 ts_tablet_manager.cc:1905] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:22.773876 13265 tablet_bootstrap.cc:492] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c: Bootstrap starting.
I20260807 02:22:22.774005 13266 tablet_bootstrap.cc:492] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190: No bootstrap required, opened a new log
I20260807 02:22:22.774083 13266 ts_tablet_manager.cc:1397] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260807 02:22:22.774539 13266 raft_consensus.cc:348] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:22.774665 13266 raft_consensus.cc:374] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:22.774708 13266 raft_consensus.cc:729] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b323f506ba084d9c84bcdd79fa644190, State: Initialized, Role: FOLLOWER
I20260807 02:22:22.774807 13288 ts_tablet_manager.cc:1918] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:22.774896 13265 tablet_bootstrap.cc:654] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:22.774842 13266 consensus_queue.cc:260] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:22.774935 13288 log.cc:1196] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/01f71383cefd4acfbcead068110b51c7
I20260807 02:22:22.775168 13266 ts_tablet_manager.cc:1428] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 02:22:22.775250 13288 ts_tablet_manager.cc:1939] T 01f71383cefd4acfbcead068110b51c7 P b323f506ba084d9c84bcdd79fa644190: Deleting consensus metadata
I20260807 02:22:22.775552 12720 catalog_manager.cc:4841] TS 7409551e595b4e609349f7360221932c (127.4.221.65:33021): tablet 01f71383cefd4acfbcead068110b51c7 (table test-table [id=b597f3111c78419ba69556144f2ff38f]) successfully deleted
I20260807 02:22:22.777098 12719 catalog_manager.cc:4841] TS 026602797d4341d8aa58fcbbd125db2e (127.4.221.67:44083): tablet 01f71383cefd4acfbcead068110b51c7 (table test-table [id=b597f3111c78419ba69556144f2ff38f]) successfully deleted
I20260807 02:22:22.777452 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 39cff190bc37421cb2e189947dda9209 (DEFAULT_TABLE table=test-table [id=5cab28bb907742b6bee3c5f5d228b2de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:22.777746 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39cff190bc37421cb2e189947dda9209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:22.780117 13264 tablet_bootstrap.cc:492] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e: Bootstrap starting.
I20260807 02:22:22.780385 12720 catalog_manager.cc:4841] TS b323f506ba084d9c84bcdd79fa644190 (127.4.221.66:32967): tablet 01f71383cefd4acfbcead068110b51c7 (table test-table [id=b597f3111c78419ba69556144f2ff38f]) successfully deleted
I20260807 02:22:22.781173 13264 tablet_bootstrap.cc:654] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:22.781279 13265 tablet_bootstrap.cc:492] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c: No bootstrap required, opened a new log
I20260807 02:22:22.781363 13265 ts_tablet_manager.cc:1397] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260807 02:22:22.781867 13265 raft_consensus.cc:348] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:22.781993 13265 raft_consensus.cc:374] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:22.782032 13265 raft_consensus.cc:729] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7409551e595b4e609349f7360221932c, State: Initialized, Role: FOLLOWER
I20260807 02:22:22.782212 13265 consensus_queue.cc:260] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:22.782400 13264 tablet_bootstrap.cc:492] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e: No bootstrap required, opened a new log
I20260807 02:22:22.782485 13264 ts_tablet_manager.cc:1397] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 02:22:22.782569 13265 ts_tablet_manager.cc:1428] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260807 02:22:22.782939 13264 raft_consensus.cc:348] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:22.783056 13264 raft_consensus.cc:374] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:22.783090 13264 raft_consensus.cc:729] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026602797d4341d8aa58fcbbd125db2e, State: Initialized, Role: FOLLOWER
I20260807 02:22:22.783237 13264 consensus_queue.cc:260] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:22.784880 13264 ts_tablet_manager.cc:1428] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260807 02:22:22.821394 13236 tablet.cc:2369] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:22:22.829442 13091 tablet.cc:2369] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:22:22.839568 12944 tablet.cc:2369] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:22:22.874158 13270 raft_consensus.cc:482] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:22:22.874325 13270 raft_consensus.cc:504] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:22.875629 13270 leader_election.cc:288] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7409551e595b4e609349f7360221932c (127.4.221.65:33021), 026602797d4341d8aa58fcbbd125db2e (127.4.221.67:44083)
I20260807 02:22:22.880353 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.875976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:49667 (local address 127.4.221.65:33021)
0807 02:22:22.876119 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.876125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.876397 (+ 272us) server_negotiation.cc:407] Connection header received
0807 02:22:22.876639 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.876644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.876703 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.876744 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:22.877158 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.877968 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.879334 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.879752 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.879892 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.879957 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.880137 (+ 180us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.880224 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 02:22:22.880378 13295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.875874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33021 (local address 127.4.221.66:49667)
0807 02:22:22.876302 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:22.876318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:22.876507 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:22.876905 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:22.876907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:22.876919 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:22.877039 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.877044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.878098 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.878102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.879209 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.879219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.879887 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.879890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.879988 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.880026 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 02:22:22.880117 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 02:22:22.880203 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":32,"spinlock_wait_cycles":19456,"thread_start_us":91,"threads_started":1}
I20260807 02:22:22.881044 12896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39cff190bc37421cb2e189947dda9209" candidate_uuid: "b323f506ba084d9c84bcdd79fa644190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7409551e595b4e609349f7360221932c" is_pre_election: true
I20260807 02:22:22.881352 12896 raft_consensus.cc:2454] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b323f506ba084d9c84bcdd79fa644190 in term 0.
I20260807 02:22:22.881594 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.876364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:58369 (local address 127.4.221.67:44083)
0807 02:22:22.876494 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.876499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.876516 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:22:22.876650 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.876653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.876710 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.876811 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:22.877941 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.878743 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.880259 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.880898 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.881018 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.881073 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.881369 (+ 296us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.881460 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 02:22:22.881842 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.875959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:44083 (local address 127.4.221.66:58369)
0807 02:22:22.876399 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:22.876412 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:22.876507 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:22.876845 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:22.876851 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:22.876875 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:22.877021 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.877029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.878894 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.878897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.880066 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.880079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.881000 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.881004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.881110 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.881134 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 02:22:22.881330 (+ 196us) client_negotiation.cc:240] Negotiation successful
0807 02:22:22.881421 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":2,"spinlock_wait_cycles":14336,"thread_start_us":71,"threads_started":1}
I20260807 02:22:22.881855 12971 leader_election.cc:302] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7409551e595b4e609349f7360221932c, b323f506ba084d9c84bcdd79fa644190; no voters:
I20260807 02:22:22.882200 13270 raft_consensus.cc:2781] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:22:22.882289 13270 raft_consensus.cc:482] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:22:22.882333 13270 raft_consensus.cc:3037] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:22.882365 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39cff190bc37421cb2e189947dda9209" candidate_uuid: "b323f506ba084d9c84bcdd79fa644190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026602797d4341d8aa58fcbbd125db2e" is_pre_election: true
I20260807 02:22:22.882525 13187 raft_consensus.cc:2454] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b323f506ba084d9c84bcdd79fa644190 in term 0.
I20260807 02:22:22.883541 13270 raft_consensus.cc:504] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:22.883965 13270 leader_election.cc:288] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [CANDIDATE]: Term 1 election: Requested vote from peers 7409551e595b4e609349f7360221932c (127.4.221.65:33021), 026602797d4341d8aa58fcbbd125db2e (127.4.221.67:44083)
I20260807 02:22:22.884382 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39cff190bc37421cb2e189947dda9209" candidate_uuid: "b323f506ba084d9c84bcdd79fa644190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026602797d4341d8aa58fcbbd125db2e"
I20260807 02:22:22.884429 12896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39cff190bc37421cb2e189947dda9209" candidate_uuid: "b323f506ba084d9c84bcdd79fa644190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7409551e595b4e609349f7360221932c"
I20260807 02:22:22.884524 13187 raft_consensus.cc:3037] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:22.884584 12896 raft_consensus.cc:3037] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:22.885504 13187 raft_consensus.cc:2454] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b323f506ba084d9c84bcdd79fa644190 in term 1.
I20260807 02:22:22.885855 12970 leader_election.cc:302] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026602797d4341d8aa58fcbbd125db2e, b323f506ba084d9c84bcdd79fa644190; no voters:
I20260807 02:22:22.886029 12896 raft_consensus.cc:2454] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b323f506ba084d9c84bcdd79fa644190 in term 1.
I20260807 02:22:22.886063 13270 raft_consensus.cc:2781] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:22.886325 13270 raft_consensus.cc:686] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [term 1 LEADER]: Becoming Leader. State: Replica: b323f506ba084d9c84bcdd79fa644190, State: Running, Role: LEADER
I20260807 02:22:22.886610 13270 consensus_queue.cc:237] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:22.889278 12731 catalog_manager.cc:5495] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 reported cstate change: term changed from 0 to 1, leader changed from <none> to b323f506ba084d9c84bcdd79fa644190 (127.4.221.66). New cstate: current_term: 1 leader_uuid: "b323f506ba084d9c84bcdd79fa644190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } health_report { overall_health: UNKNOWN } } }
Aug 07 02:22:22 dist-test-slave-69rg krb5kdc[12689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069341, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.126@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
I20260807 02:22:22.956964 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.947924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50090 (local address 127.4.221.126:43993)
0807 02:22:22.948108 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:22.948113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:22.948322 (+ 209us) server_negotiation.cc:407] Connection header received
0807 02:22:22.948490 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:22.948493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:22.948538 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:22.948647 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:22.949331 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.949946 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.950725 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.950942 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.951054 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.953529 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:22.953533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:22.953536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:22.953566 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:22.955003 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.955549 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.955554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.955556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.955636 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:22.955922 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:22.955925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:22.955927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:22.956158 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:22.956272 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:22.956575 (+ 303us) server_negotiation.cc:299] Negotiation successful
0807 02:22:22.956755 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":67}
I20260807 02:22:22.956982 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:22.947852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:43993 (local address 127.0.0.1:50090)
0807 02:22:22.948225 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:22.948237 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:22.948390 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:22.948673 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:22.948676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:22.949111 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:22.949208 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.949215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.950052 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.950058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.950601 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:22.950608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:22.951059 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:22.951063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:22.951143 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:22.951762 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:22.951774 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:22.953371 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:22.955154 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:22.955167 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:22.955175 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:22.955442 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:22.955740 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:22.955743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:22.955745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:22.955825 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:22.956271 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:22.956280 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:22.956403 (+ 123us) client_negotiation.cc:769] Sending connection context
0807 02:22:22.956530 (+ 127us) client_negotiation.cc:240] Negotiation successful
0807 02:22:22.956818 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":118,"thread_start_us":141,"threads_started":1}
I20260807 02:22:22.961633 12731 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:50090:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260807 02:22:22.962528 12731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:22:22.968189 13021 tablet_service.cc:1467] Processing CreateTablet for tablet d028a94614134c64938c63140eb7e71e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed3e9faa4dc5429f9f0018bb371d5960]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:22:22.968192 12875 tablet_service.cc:1467] Processing CreateTablet for tablet d028a94614134c64938c63140eb7e71e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed3e9faa4dc5429f9f0018bb371d5960]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:22:22.968192 13166 tablet_service.cc:1467] Processing CreateTablet for tablet d028a94614134c64938c63140eb7e71e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed3e9faa4dc5429f9f0018bb371d5960]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260807 02:22:22.968633 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d028a94614134c64938c63140eb7e71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:22.968633 12875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d028a94614134c64938c63140eb7e71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:22.968648 13021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d028a94614134c64938c63140eb7e71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:22.971530 13266 tablet_bootstrap.cc:492] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190: Bootstrap starting.
I20260807 02:22:22.971561 13265 tablet_bootstrap.cc:492] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c: Bootstrap starting.
I20260807 02:22:22.971588 13264 tablet_bootstrap.cc:492] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e: Bootstrap starting.
I20260807 02:22:22.972507 13265 tablet_bootstrap.cc:654] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:22.972512 13264 tablet_bootstrap.cc:654] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:22.972666 13266 tablet_bootstrap.cc:654] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:22.973759 13264 tablet_bootstrap.cc:492] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e: No bootstrap required, opened a new log
I20260807 02:22:22.973882 13264 ts_tablet_manager.cc:1397] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 02:22:22.973910 13265 tablet_bootstrap.cc:492] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c: No bootstrap required, opened a new log
I20260807 02:22:22.973982 13265 ts_tablet_manager.cc:1397] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 02:22:22.974212 13266 tablet_bootstrap.cc:492] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190: No bootstrap required, opened a new log
I20260807 02:22:22.974295 13266 ts_tablet_manager.cc:1397] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 02:22:22.974354 13264 raft_consensus.cc:348] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:22.974354 13265 raft_consensus.cc:348] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:22.974467 13265 raft_consensus.cc:374] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:22.974468 13264 raft_consensus.cc:374] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:22.974502 13265 raft_consensus.cc:729] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7409551e595b4e609349f7360221932c, State: Initialized, Role: FOLLOWER
I20260807 02:22:22.974502 13264 raft_consensus.cc:729] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026602797d4341d8aa58fcbbd125db2e, State: Initialized, Role: FOLLOWER
I20260807 02:22:22.974627 13264 consensus_queue.cc:260] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:22.974627 13265 consensus_queue.cc:260] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:22.974761 13266 raft_consensus.cc:348] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:22.974884 13266 raft_consensus.cc:374] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:22.974887 13265 ts_tablet_manager.cc:1428] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 02:22:22.974887 13264 ts_tablet_manager.cc:1428] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 02:22:22.974924 13266 raft_consensus.cc:729] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b323f506ba084d9c84bcdd79fa644190, State: Initialized, Role: FOLLOWER
I20260807 02:22:22.975085 13266 consensus_queue.cc:260] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:22.975117 13284 tablet_replica.cc:439] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } }
I20260807 02:22:22.975155 13271 tablet_replica.cc:439] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } }
I20260807 02:22:22.975332 13284 tablet_replica.cc:442] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:22:22.975337 13271 tablet_replica.cc:442] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:22:22.975328 13270 tablet_replica.cc:439] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } }
I20260807 02:22:22.975459 13270 tablet_replica.cc:442] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:22:22.975670 13266 ts_tablet_manager.cc:1428] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260807 02:22:23.072604 13284 raft_consensus.cc:482] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:22:23.072762 13284 raft_consensus.cc:504] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.073302 13284 leader_election.cc:288] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b323f506ba084d9c84bcdd79fa644190 (127.4.221.66:32967), 026602797d4341d8aa58fcbbd125db2e (127.4.221.67:44083)
I20260807 02:22:23.073855 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d028a94614134c64938c63140eb7e71e" candidate_uuid: "7409551e595b4e609349f7360221932c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026602797d4341d8aa58fcbbd125db2e" is_pre_election: true
I20260807 02:22:23.073850 13042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d028a94614134c64938c63140eb7e71e" candidate_uuid: "7409551e595b4e609349f7360221932c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b323f506ba084d9c84bcdd79fa644190" is_pre_election: true
I20260807 02:22:23.074018 13187 raft_consensus.cc:2454] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7409551e595b4e609349f7360221932c in term 0.
I20260807 02:22:23.074018 13042 raft_consensus.cc:2454] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7409551e595b4e609349f7360221932c in term 0.
I20260807 02:22:23.074391 12827 leader_election.cc:302] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7409551e595b4e609349f7360221932c, b323f506ba084d9c84bcdd79fa644190; no voters:
I20260807 02:22:23.074644 13284 raft_consensus.cc:2781] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:22:23.074725 13284 raft_consensus.cc:482] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:22:23.074755 13284 raft_consensus.cc:3037] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:23.075929 13284 raft_consensus.cc:504] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.076341 13284 leader_election.cc:288] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [CANDIDATE]: Term 1 election: Requested vote from peers b323f506ba084d9c84bcdd79fa644190 (127.4.221.66:32967), 026602797d4341d8aa58fcbbd125db2e (127.4.221.67:44083)
I20260807 02:22:23.076861 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d028a94614134c64938c63140eb7e71e" candidate_uuid: "7409551e595b4e609349f7360221932c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026602797d4341d8aa58fcbbd125db2e"
I20260807 02:22:23.076901 13042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d028a94614134c64938c63140eb7e71e" candidate_uuid: "7409551e595b4e609349f7360221932c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b323f506ba084d9c84bcdd79fa644190"
I20260807 02:22:23.076989 13187 raft_consensus.cc:3037] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:23.077095 13042 raft_consensus.cc:3037] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:23.078169 13187 raft_consensus.cc:2454] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7409551e595b4e609349f7360221932c in term 1.
I20260807 02:22:23.078487 13042 raft_consensus.cc:2454] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7409551e595b4e609349f7360221932c in term 1.
I20260807 02:22:23.078706 12826 leader_election.cc:302] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026602797d4341d8aa58fcbbd125db2e, 7409551e595b4e609349f7360221932c; no voters:
I20260807 02:22:23.078971 13284 raft_consensus.cc:2781] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:23.079056 13284 raft_consensus.cc:686] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [term 1 LEADER]: Becoming Leader. State: Replica: 7409551e595b4e609349f7360221932c, State: Running, Role: LEADER
I20260807 02:22:23.079260 13284 consensus_queue.cc:237] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.080049 13272 tablet_replica.cc:439] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c: TxnStatusTablet state changed. Reason: New leader 7409551e595b4e609349f7360221932c. Latest consensus state: current_term: 1 leader_uuid: "7409551e595b4e609349f7360221932c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } }
I20260807 02:22:23.080207 13272 tablet_replica.cc:442] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:22:23.080598 13308 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260807 02:22:23.080793 13308 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260807 02:22:23.081282 12734 catalog_manager.cc:5495] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7409551e595b4e609349f7360221932c (127.4.221.65). New cstate: current_term: 1 leader_uuid: "7409551e595b4e609349f7360221932c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20260807 02:22:23.102454 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:23.099146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59778 (local address 127.4.221.65:33021)
0807 02:22:23.099293 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:23.099299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:23.099313 (+ 14us) server_negotiation.cc:407] Connection header received
0807 02:22:23.099369 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:23.099371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:23.099412 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:23.099497 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:22:23.099787 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.100326 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.101233 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.101467 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.101580 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:23.101940 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:22:23.102026 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:22:23.102106 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:23.102246 (+ 140us) server_negotiation.cc:299] Negotiation successful
0807 02:22:23.102301 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 02:22:23.102454 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:23.098887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33021 (local address 127.0.0.1:59778)
0807 02:22:23.099041 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:23.099058 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:23.099181 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:23.099531 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:23.099534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:23.099542 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:22:23.099663 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:23.099668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.100455 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.100459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:23.101097 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:23.101106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.101587 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.101592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:23.101696 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:23.101752 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:22:23.102124 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:22:23.102141 (+ 17us) client_negotiation.cc:769] Sending connection context
0807 02:22:23.102228 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 02:22:23.102324 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260807 02:22:23.105499 13042 raft_consensus.cc:1261] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Refusing update from remote peer 7409551e595b4e609349f7360221932c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:23.106030 13272 consensus_queue.cc:1035] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:23.106247 13187 raft_consensus.cc:1261] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Refusing update from remote peer 7409551e595b4e609349f7360221932c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:23.106798 13272 consensus_queue.cc:1035] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c [LEADER]: Connected to new peer: Peer: permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:23.107856 13270 tablet_replica.cc:439] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190: TxnStatusTablet state changed. Reason: New leader 7409551e595b4e609349f7360221932c. Latest consensus state: current_term: 1 leader_uuid: "7409551e595b4e609349f7360221932c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } }
I20260807 02:22:23.107975 13270 tablet_replica.cc:442] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:22:23.108094 13271 tablet_replica.cc:439] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e: TxnStatusTablet state changed. Reason: New leader 7409551e595b4e609349f7360221932c. Latest consensus state: current_term: 1 leader_uuid: "7409551e595b4e609349f7360221932c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } }
I20260807 02:22:23.108250 13271 tablet_replica.cc:442] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:22:23.110100 13284 tablet_replica.cc:439] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7409551e595b4e609349f7360221932c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } }
I20260807 02:22:23.110229 13284 tablet_replica.cc:442] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:22:23.111421 13271 tablet_replica.cc:439] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7409551e595b4e609349f7360221932c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } }
I20260807 02:22:23.111521 13271 tablet_replica.cc:442] T d028a94614134c64938c63140eb7e71e P 026602797d4341d8aa58fcbbd125db2e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:22:23.114384 13284 tablet_replica.cc:439] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7409551e595b4e609349f7360221932c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } }
I20260807 02:22:23.114519 13284 tablet_replica.cc:442] T d028a94614134c64938c63140eb7e71e P 7409551e595b4e609349f7360221932c: This TxnStatusTablet replica's current role is: LEADER
I20260807 02:22:23.115633 13270 tablet_replica.cc:439] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7409551e595b4e609349f7360221932c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } } }
I20260807 02:22:23.115777 13270 tablet_replica.cc:442] T d028a94614134c64938c63140eb7e71e P b323f506ba084d9c84bcdd79fa644190: This TxnStatusTablet replica's current role is: FOLLOWER
I20260807 02:22:23.121145 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:23.117274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35204 (local address 127.4.221.66:32967)
0807 02:22:23.117403 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:23.117407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:23.117420 (+ 13us) server_negotiation.cc:407] Connection header received
0807 02:22:23.117505 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:23.117507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:23.117576 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:23.117702 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:22:23.118082 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.118606 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.119589 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.119828 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.119940 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:23.120286 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:22:23.120754 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:22:23.120884 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:23.120948 (+ 64us) server_negotiation.cc:299] Negotiation successful
0807 02:22:23.121005 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 02:22:23.132552 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:23.129117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:55153 (local address 127.4.221.65:33021)
0807 02:22:23.129291 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:23.129296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:23.129396 (+ 100us) server_negotiation.cc:407] Connection header received
0807 02:22:23.129558 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:23.129561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:23.129621 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:23.129705 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:22:23.129995 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.130520 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.131397 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.131645 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.131763 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:23.132071 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:22:23.132154 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:22:23.132239 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:23.132329 (+ 90us) server_negotiation.cc:299] Negotiation successful
0807 02:22:23.132388 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 02:22:23.133114 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:23.128981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33021 (local address 127.4.221.66:55153)
0807 02:22:23.129302 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:23.129318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:23.129455 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:23.129723 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:23.129726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:23.129734 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:22:23.129883 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:23.129890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.130628 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.130634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:23.131277 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:23.131286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.131779 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.131783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:23.131879 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:23.131925 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:22:23.132244 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:22:23.132253 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 02:22:23.132319 (+ 66us) client_negotiation.cc:240] Negotiation successful
0807 02:22:23.132403 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260807 02:22:23.141036 13187 raft_consensus.cc:1261] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Refusing update from remote peer b323f506ba084d9c84bcdd79fa644190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:23.141343 12896 raft_consensus.cc:1261] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Refusing update from remote peer b323f506ba084d9c84bcdd79fa644190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:23.141552 13270 consensus_queue.cc:1035] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:23.141934 13296 consensus_queue.cc:1035] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:23.147946 13280 mvcc.cc:205] Tried to move back new op lower bound from 7315740029501009920 to 7315740028467335168. Current Snapshot: MvccSnapshot[applied={T|T < 7315740029501009920}]
I20260807 02:22:23.168385 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:23.164040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:49255 (local address 127.4.221.66:32967)
0807 02:22:23.164184 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:23.164189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:23.164439 (+ 250us) server_negotiation.cc:407] Connection header received
0807 02:22:23.164543 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:23.164550 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:23.164635 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:23.164723 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:22:23.164991 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.165721 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.167196 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.167509 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.167607 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:23.167920 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:22:23.168004 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:22:23.168086 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:23.168175 (+ 89us) server_negotiation.cc:299] Negotiation successful
0807 02:22:23.168237 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:22:23.168396 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:23.163961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:32967 (local address 127.4.221.65:49255)
0807 02:22:23.164326 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:23.164343 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:23.164454 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:23.164741 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:23.164744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:23.164752 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0807 02:22:23.164866 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:23.164872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.165876 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.165880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:23.167040 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:23.167051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.167618 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.167622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:23.167714 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:23.167769 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0807 02:22:23.168090 (+ 321us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0807 02:22:23.168099 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 02:22:23.168166 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 02:22:23.168265 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":138,"threads_started":1}
I20260807 02:22:23.194187 12734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:23.194427 12734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:23.196789 12734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249: Sending DeleteTablet for 3 replicas of tablet 39cff190bc37421cb2e189947dda9209
I20260807 02:22:23.197461 12875 tablet_service.cc:1514] Processing DeleteTablet for tablet 39cff190bc37421cb2e189947dda9209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:23 UTC) from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:59766
I20260807 02:22:23.197657 13021 tablet_service.cc:1514] Processing DeleteTablet for tablet 39cff190bc37421cb2e189947dda9209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:23 UTC) from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:35190
I20260807 02:22:23.197711 13290 tablet_replica.cc:330] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c: stopping tablet replica
I20260807 02:22:23.197865 13288 tablet_replica.cc:330] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190: stopping tablet replica
I20260807 02:22:23.197873 13290 raft_consensus.cc:2229] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:23.197978 13288 raft_consensus.cc:2229] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:22:23.198022 13290 raft_consensus.cc:2258] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:23.198347 13288 raft_consensus.cc:2258] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:23.199162 13290 ts_tablet_manager.cc:1905] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:23.199162 13288 ts_tablet_manager.cc:1905] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:23.202198 13288 ts_tablet_manager.cc:1918] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:22:23.202198 13290 ts_tablet_manager.cc:1918] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:22:23.202293 13290 log.cc:1196] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/wal/wals/39cff190bc37421cb2e189947dda9209
I20260807 02:22:23.202293 13288 log.cc:1196] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/39cff190bc37421cb2e189947dda9209
I20260807 02:22:23.202582 13288 ts_tablet_manager.cc:1939] T 39cff190bc37421cb2e189947dda9209 P b323f506ba084d9c84bcdd79fa644190: Deleting consensus metadata
I20260807 02:22:23.202589 13290 ts_tablet_manager.cc:1939] T 39cff190bc37421cb2e189947dda9209 P 7409551e595b4e609349f7360221932c: Deleting consensus metadata
I20260807 02:22:23.203477 13166 tablet_service.cc:1514] Processing DeleteTablet for tablet 39cff190bc37421cb2e189947dda9209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:23 UTC) from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:54928
I20260807 02:22:23.203476 12720 catalog_manager.cc:4841] TS 7409551e595b4e609349f7360221932c (127.4.221.65:33021): tablet 39cff190bc37421cb2e189947dda9209 (table test-table [id=5cab28bb907742b6bee3c5f5d228b2de]) successfully deleted
I20260807 02:22:23.203722 13289 tablet_replica.cc:330] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e: stopping tablet replica
I20260807 02:22:23.203867 13289 raft_consensus.cc:2229] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:23.204015 13289 raft_consensus.cc:2258] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:23.204936 13289 ts_tablet_manager.cc:1905] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:23.205135 12720 catalog_manager.cc:4841] TS b323f506ba084d9c84bcdd79fa644190 (127.4.221.66:32967): tablet 39cff190bc37421cb2e189947dda9209 (table test-table [id=5cab28bb907742b6bee3c5f5d228b2de]) successfully deleted
I20260807 02:22:23.207449 13289 ts_tablet_manager.cc:1918] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:22:23.207563 13289 log.cc:1196] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/39cff190bc37421cb2e189947dda9209
I20260807 02:22:23.207813 13289 ts_tablet_manager.cc:1939] T 39cff190bc37421cb2e189947dda9209 P 026602797d4341d8aa58fcbbd125db2e: Deleting consensus metadata
I20260807 02:22:23.209079 12719 catalog_manager.cc:4841] TS 026602797d4341d8aa58fcbbd125db2e (127.4.221.67:44083): tablet 39cff190bc37421cb2e189947dda9209 (table test-table [id=5cab28bb907742b6bee3c5f5d228b2de]) successfully deleted
Aug 07 02:22:23 dist-test-slave-69rg krb5kdc[12689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 07 02:22:23 dist-test-slave-69rg krb5kdc[12689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:23 dist-test-slave-69rg krb5kdc[12689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 02:22:23.236135 12784 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:23.229321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50092 (local address 127.4.221.126:43993)
0807 02:22:23.229470 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:23.229475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:23.229729 (+ 254us) server_negotiation.cc:407] Connection header received
0807 02:22:23.229938 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:23.229943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:23.229995 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:23.230084 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:23.230819 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.231364 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.232237 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.232471 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.232585 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:23.235992 (+ 3407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50092: BlockingRecv error: recv got EOF from 127.0.0.1:50092 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 07 02:22:23 dist-test-slave-69rg krb5kdc[12689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM
I20260807 02:22:23.247237 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:23.238152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.4.221.126:43993)
0807 02:22:23.238305 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:23.238310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:23.238713 (+ 403us) server_negotiation.cc:407] Connection header received
0807 02:22:23.238898 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:23.238903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:23.238955 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:23.239051 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:23.239753 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.240287 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.241082 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.241320 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.241426 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:23.243923 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:23.243927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:23.243929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:23.243965 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:23.245470 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:23.245974 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:23.245979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:23.245981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:23.246064 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:23.246354 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:23.246358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:23.246360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:23.246553 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:23.246696 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:23.246942 (+ 246us) server_negotiation.cc:299] Negotiation successful
0807 02:22:23.247065 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 02:22:23.250308 12734 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:22:23.250795 12734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:22:23.255637 12875 tablet_service.cc:1467] Processing CreateTablet for tablet f8975656c51442eca2576ebd45e450f8 (DEFAULT_TABLE table=test-table [id=1ad619a30449475aa79b5e3026a14e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:23.255767 13166 tablet_service.cc:1467] Processing CreateTablet for tablet f8975656c51442eca2576ebd45e450f8 (DEFAULT_TABLE table=test-table [id=1ad619a30449475aa79b5e3026a14e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:23.255990 12875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8975656c51442eca2576ebd45e450f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:23.256018 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8975656c51442eca2576ebd45e450f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:23.258412 13021 tablet_service.cc:1467] Processing CreateTablet for tablet f8975656c51442eca2576ebd45e450f8 (DEFAULT_TABLE table=test-table [id=1ad619a30449475aa79b5e3026a14e2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:23.258725 13021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8975656c51442eca2576ebd45e450f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:23.260753 13266 tablet_bootstrap.cc:492] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190: Bootstrap starting.
I20260807 02:22:23.260869 13264 tablet_bootstrap.cc:492] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e: Bootstrap starting.
I20260807 02:22:23.261756 13266 tablet_bootstrap.cc:654] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:23.261766 13264 tablet_bootstrap.cc:654] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:23.263815 13265 tablet_bootstrap.cc:492] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c: Bootstrap starting.
I20260807 02:22:23.264783 13265 tablet_bootstrap.cc:654] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:23.265031 13264 tablet_bootstrap.cc:492] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e: No bootstrap required, opened a new log
I20260807 02:22:23.265110 13264 ts_tablet_manager.cc:1397] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 02:22:23.265652 13264 raft_consensus.cc:348] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.266175 13264 raft_consensus.cc:374] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:23.266237 13264 raft_consensus.cc:729] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026602797d4341d8aa58fcbbd125db2e, State: Initialized, Role: FOLLOWER
I20260807 02:22:23.266304 13265 tablet_bootstrap.cc:492] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c: No bootstrap required, opened a new log
I20260807 02:22:23.266382 13265 ts_tablet_manager.cc:1397] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 02:22:23.266373 13264 consensus_queue.cc:260] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.266662 13264 ts_tablet_manager.cc:1428] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 02:22:23.266798 13265 raft_consensus.cc:348] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.266909 13265 raft_consensus.cc:374] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:23.266947 13265 raft_consensus.cc:729] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7409551e595b4e609349f7360221932c, State: Initialized, Role: FOLLOWER
I20260807 02:22:23.267091 13265 consensus_queue.cc:260] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.267390 13265 ts_tablet_manager.cc:1428] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 02:22:23.270628 13266 tablet_bootstrap.cc:492] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190: No bootstrap required, opened a new log
I20260807 02:22:23.270766 13266 ts_tablet_manager.cc:1397] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20260807 02:22:23.271210 13266 raft_consensus.cc:348] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.271350 13266 raft_consensus.cc:374] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:23.271392 13266 raft_consensus.cc:729] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b323f506ba084d9c84bcdd79fa644190, State: Initialized, Role: FOLLOWER
I20260807 02:22:23.271541 13266 consensus_queue.cc:260] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.271823 13266 ts_tablet_manager.cc:1428] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 02:22:23.419762 13271 raft_consensus.cc:482] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:22:23.419924 13271 raft_consensus.cc:504] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.421010 13271 leader_election.cc:288] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7409551e595b4e609349f7360221932c (127.4.221.65:33021), b323f506ba084d9c84bcdd79fa644190 (127.4.221.66:32967)
I20260807 02:22:23.426095 13263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:23.421506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:54869 (local address 127.4.221.65:33021)
0807 02:22:23.421682 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:23.421688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:23.421932 (+ 244us) server_negotiation.cc:407] Connection header received
0807 02:22:23.422129 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:23.422132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:23.422190 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:23.422269 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:23.422668 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.423503 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.424875 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.425505 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.425623 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:23.425678 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:23.425898 (+ 220us) server_negotiation.cc:299] Negotiation successful
0807 02:22:23.425985 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 02:22:23.426128 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:23.421382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33021 (local address 127.4.221.67:54869)
0807 02:22:23.421773 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:23.421788 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:23.421995 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:23.422309 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:23.422312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:23.422332 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:23.422510 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:23.422517 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.423634 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.423637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:23.424703 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:23.424739 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.425625 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.425629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:23.425718 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:23.425746 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 02:22:23.425858 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 02:22:23.425935 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"spinlock_wait_cycles":1792,"thread_start_us":136,"threads_started":1}
I20260807 02:22:23.426801 12896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8975656c51442eca2576ebd45e450f8" candidate_uuid: "026602797d4341d8aa58fcbbd125db2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7409551e595b4e609349f7360221932c" is_pre_election: true
I20260807 02:22:23.426954 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:23.421924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:32967 (local address 127.4.221.67:55143)
0807 02:22:23.422285 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:23.422299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:23.422408 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:23.422698 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:23.422701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:23.422721 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:23.423276 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:23.423286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.424278 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.424282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:23.425368 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:23.425379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.426562 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.426568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:23.426670 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:23.426699 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 02:22:23.426763 (+ 64us) client_negotiation.cc:240] Negotiation successful
0807 02:22:23.426815 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"spinlock_wait_cycles":22784,"thread_start_us":72,"threads_started":1}
I20260807 02:22:23.426978 12896 raft_consensus.cc:2454] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 026602797d4341d8aa58fcbbd125db2e in term 0.
I20260807 02:22:23.427014 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:23.422022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:55143 (local address 127.4.221.66:32967)
0807 02:22:23.422160 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:23.422163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:23.422379 (+ 216us) server_negotiation.cc:407] Connection header received
0807 02:22:23.422518 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:23.422521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:23.422585 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:23.422661 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:23.423406 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.424142 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.425511 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.426154 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.426297 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:23.426361 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:23.426814 (+ 453us) server_negotiation.cc:299] Negotiation successful
0807 02:22:23.426900 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 02:22:23.427487 13042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8975656c51442eca2576ebd45e450f8" candidate_uuid: "026602797d4341d8aa58fcbbd125db2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b323f506ba084d9c84bcdd79fa644190" is_pre_election: true
I20260807 02:22:23.427567 13119 leader_election.cc:302] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026602797d4341d8aa58fcbbd125db2e, 7409551e595b4e609349f7360221932c; no voters:
I20260807 02:22:23.427656 13042 raft_consensus.cc:2454] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 026602797d4341d8aa58fcbbd125db2e in term 0.
I20260807 02:22:23.427927 13271 raft_consensus.cc:2781] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:22:23.427987 13271 raft_consensus.cc:482] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:22:23.428047 13271 raft_consensus.cc:3037] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:23.429122 13271 raft_consensus.cc:504] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.429491 13271 leader_election.cc:288] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [CANDIDATE]: Term 1 election: Requested vote from peers 7409551e595b4e609349f7360221932c (127.4.221.65:33021), b323f506ba084d9c84bcdd79fa644190 (127.4.221.66:32967)
I20260807 02:22:23.429975 12896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8975656c51442eca2576ebd45e450f8" candidate_uuid: "026602797d4341d8aa58fcbbd125db2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7409551e595b4e609349f7360221932c"
I20260807 02:22:23.430182 12896 raft_consensus.cc:3037] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:23.430097 13042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8975656c51442eca2576ebd45e450f8" candidate_uuid: "026602797d4341d8aa58fcbbd125db2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b323f506ba084d9c84bcdd79fa644190"
I20260807 02:22:23.430318 13042 raft_consensus.cc:3037] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:23.431242 12896 raft_consensus.cc:2454] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 026602797d4341d8aa58fcbbd125db2e in term 1.
I20260807 02:22:23.431268 13042 raft_consensus.cc:2454] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 026602797d4341d8aa58fcbbd125db2e in term 1.
I20260807 02:22:23.431545 13119 leader_election.cc:302] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 026602797d4341d8aa58fcbbd125db2e, b323f506ba084d9c84bcdd79fa644190; no voters:
I20260807 02:22:23.431746 13271 raft_consensus.cc:2781] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:23.431950 13271 raft_consensus.cc:686] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [term 1 LEADER]: Becoming Leader. State: Replica: 026602797d4341d8aa58fcbbd125db2e, State: Running, Role: LEADER
I20260807 02:22:23.432237 13271 consensus_queue.cc:237] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.434551 12734 catalog_manager.cc:5495] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 026602797d4341d8aa58fcbbd125db2e (127.4.221.67). New cstate: current_term: 1 leader_uuid: "026602797d4341d8aa58fcbbd125db2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } health_report { overall_health: HEALTHY } } }
I20260807 02:22:23.472625 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:23.468697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54932 (local address 127.4.221.67:44083)
0807 02:22:23.469045 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:23.469053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:23.469065 (+ 12us) server_negotiation.cc:407] Connection header received
0807 02:22:23.469150 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:23.469153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:23.469209 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:23.469354 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:22:23.469676 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.470263 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.471128 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.471367 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.471443 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:23.471757 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:22:23.472242 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:22:23.472328 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:23.472433 (+ 105us) server_negotiation.cc:299] Negotiation successful
0807 02:22:23.472485 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":153,"threads_started":1}
I20260807 02:22:23.475558 13042 raft_consensus.cc:1261] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Refusing update from remote peer 026602797d4341d8aa58fcbbd125db2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:23.475564 12896 raft_consensus.cc:1261] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Refusing update from remote peer 026602797d4341d8aa58fcbbd125db2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:23.476126 13341 consensus_queue.cc:1035] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:23.476343 13271 consensus_queue.cc:1035] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:23.487914 12731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:23.488139 12731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:23.490705 12731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249: Sending DeleteTablet for 3 replicas of tablet f8975656c51442eca2576ebd45e450f8
I20260807 02:22:23.491475 13166 tablet_service.cc:1514] Processing DeleteTablet for tablet f8975656c51442eca2576ebd45e450f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:23 UTC) from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:54928
I20260807 02:22:23.491546 13021 tablet_service.cc:1514] Processing DeleteTablet for tablet f8975656c51442eca2576ebd45e450f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:23 UTC) from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:35190
I20260807 02:22:23.491712 13289 tablet_replica.cc:330] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e: stopping tablet replica
I20260807 02:22:23.491825 13288 tablet_replica.cc:330] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190: stopping tablet replica
I20260807 02:22:23.491849 13289 raft_consensus.cc:2229] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:22:23.491962 13288 raft_consensus.cc:2229] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:23.492110 13288 raft_consensus.cc:2258] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:23.492216 13289 raft_consensus.cc:2258] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:23.492254 12731 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260807 02:22:23.492631 12875 tablet_service.cc:1514] Processing DeleteTablet for tablet f8975656c51442eca2576ebd45e450f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:23 UTC) from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:59766
W20260807 02:22:23.492763 12731 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:22:23.492862 13290 tablet_replica.cc:330] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c: stopping tablet replica
I20260807 02:22:23.493005 13290 raft_consensus.cc:2229] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:23.493153 13290 raft_consensus.cc:2258] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:23.493602 13289 ts_tablet_manager.cc:1905] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:23.494107 13288 ts_tablet_manager.cc:1905] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:23.496417 13289 ts_tablet_manager.cc:1918] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:23.496496 13289 log.cc:1196] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/f8975656c51442eca2576ebd45e450f8
I20260807 02:22:23.496656 13288 ts_tablet_manager.cc:1918] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:23.496745 13288 log.cc:1196] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/f8975656c51442eca2576ebd45e450f8
I20260807 02:22:23.496789 13289 ts_tablet_manager.cc:1939] T f8975656c51442eca2576ebd45e450f8 P 026602797d4341d8aa58fcbbd125db2e: Deleting consensus metadata
I20260807 02:22:23.497010 13288 ts_tablet_manager.cc:1939] T f8975656c51442eca2576ebd45e450f8 P b323f506ba084d9c84bcdd79fa644190: Deleting consensus metadata
I20260807 02:22:23.497802 13290 ts_tablet_manager.cc:1905] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:23.498027 12719 catalog_manager.cc:4841] TS 026602797d4341d8aa58fcbbd125db2e (127.4.221.67:44083): tablet f8975656c51442eca2576ebd45e450f8 (table test-table [id=1ad619a30449475aa79b5e3026a14e2b]) successfully deleted
I20260807 02:22:23.498968 12875 tablet_service.cc:1467] Processing CreateTablet for tablet 1a650c557d13433dbedf40b9085f7c10 (DEFAULT_TABLE table=test-table [id=c2c575cce7ae42ac8b71c6b479dd4a18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:23.499276 12875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a650c557d13433dbedf40b9085f7c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:23.500386 13290 ts_tablet_manager.cc:1918] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:23.500458 13290 log.cc:1196] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-0/wal/wals/f8975656c51442eca2576ebd45e450f8
I20260807 02:22:23.500725 13290 ts_tablet_manager.cc:1939] T f8975656c51442eca2576ebd45e450f8 P 7409551e595b4e609349f7360221932c: Deleting consensus metadata
I20260807 02:22:23.501240 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 1a650c557d13433dbedf40b9085f7c10 (DEFAULT_TABLE table=test-table [id=c2c575cce7ae42ac8b71c6b479dd4a18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:23.501557 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a650c557d13433dbedf40b9085f7c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:23.501748 13265 tablet_bootstrap.cc:492] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c: Bootstrap starting.
I20260807 02:22:23.501732 13021 tablet_service.cc:1467] Processing CreateTablet for tablet 1a650c557d13433dbedf40b9085f7c10 (DEFAULT_TABLE table=test-table [id=c2c575cce7ae42ac8b71c6b479dd4a18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:23.502009 13021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a650c557d13433dbedf40b9085f7c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:23.502663 13265 tablet_bootstrap.cc:654] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:23.503955 13265 tablet_bootstrap.cc:492] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c: No bootstrap required, opened a new log
I20260807 02:22:23.504040 13265 ts_tablet_manager.cc:1397] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260807 02:22:23.504366 13266 tablet_bootstrap.cc:492] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190: Bootstrap starting.
I20260807 02:22:23.504447 13265 raft_consensus.cc:348] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.504573 13265 raft_consensus.cc:374] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:23.504608 13265 raft_consensus.cc:729] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7409551e595b4e609349f7360221932c, State: Initialized, Role: FOLLOWER
I20260807 02:22:23.504796 13264 tablet_bootstrap.cc:492] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e: Bootstrap starting.
I20260807 02:22:23.504752 13265 consensus_queue.cc:260] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.505021 13265 ts_tablet_manager.cc:1428] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 02:22:23.505196 13266 tablet_bootstrap.cc:654] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:23.505740 13264 tablet_bootstrap.cc:654] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:23.505777 12720 catalog_manager.cc:4841] TS 7409551e595b4e609349f7360221932c (127.4.221.65:33021): tablet f8975656c51442eca2576ebd45e450f8 (table test-table [id=1ad619a30449475aa79b5e3026a14e2b]) successfully deleted
I20260807 02:22:23.506057 12720 catalog_manager.cc:4841] TS b323f506ba084d9c84bcdd79fa644190 (127.4.221.66:32967): tablet f8975656c51442eca2576ebd45e450f8 (table test-table [id=1ad619a30449475aa79b5e3026a14e2b]) successfully deleted
I20260807 02:22:23.506671 13266 tablet_bootstrap.cc:492] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190: No bootstrap required, opened a new log
I20260807 02:22:23.506775 13266 ts_tablet_manager.cc:1397] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260807 02:22:23.507186 13266 raft_consensus.cc:348] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.507301 13266 raft_consensus.cc:374] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:23.507340 13266 raft_consensus.cc:729] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b323f506ba084d9c84bcdd79fa644190, State: Initialized, Role: FOLLOWER
I20260807 02:22:23.507462 13266 consensus_queue.cc:260] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.507714 13266 ts_tablet_manager.cc:1428] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260807 02:22:23.508508 13264 tablet_bootstrap.cc:492] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e: No bootstrap required, opened a new log
I20260807 02:22:23.508610 13264 ts_tablet_manager.cc:1397] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260807 02:22:23.508993 13264 raft_consensus.cc:348] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.509088 13264 raft_consensus.cc:374] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:23.509110 13264 raft_consensus.cc:729] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026602797d4341d8aa58fcbbd125db2e, State: Initialized, Role: FOLLOWER
I20260807 02:22:23.509296 13264 consensus_queue.cc:260] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.509584 13264 ts_tablet_manager.cc:1428] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260807 02:22:23.721367 13270 raft_consensus.cc:482] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:22:23.721568 13270 raft_consensus.cc:504] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.722175 13270 leader_election.cc:288] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7409551e595b4e609349f7360221932c (127.4.221.65:33021), 026602797d4341d8aa58fcbbd125db2e (127.4.221.67:44083)
I20260807 02:22:23.722764 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a650c557d13433dbedf40b9085f7c10" candidate_uuid: "b323f506ba084d9c84bcdd79fa644190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026602797d4341d8aa58fcbbd125db2e" is_pre_election: true
I20260807 02:22:23.722776 12896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a650c557d13433dbedf40b9085f7c10" candidate_uuid: "b323f506ba084d9c84bcdd79fa644190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7409551e595b4e609349f7360221932c" is_pre_election: true
I20260807 02:22:23.722942 13187 raft_consensus.cc:2454] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b323f506ba084d9c84bcdd79fa644190 in term 0.
I20260807 02:22:23.722942 12896 raft_consensus.cc:2454] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b323f506ba084d9c84bcdd79fa644190 in term 0.
I20260807 02:22:23.723315 12971 leader_election.cc:302] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7409551e595b4e609349f7360221932c, b323f506ba084d9c84bcdd79fa644190; no voters:
I20260807 02:22:23.723539 13270 raft_consensus.cc:2781] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:22:23.723610 13270 raft_consensus.cc:482] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:22:23.723639 13270 raft_consensus.cc:3037] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:23.724823 13270 raft_consensus.cc:504] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.725206 13270 leader_election.cc:288] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [CANDIDATE]: Term 1 election: Requested vote from peers 7409551e595b4e609349f7360221932c (127.4.221.65:33021), 026602797d4341d8aa58fcbbd125db2e (127.4.221.67:44083)
I20260807 02:22:23.725660 12896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a650c557d13433dbedf40b9085f7c10" candidate_uuid: "b323f506ba084d9c84bcdd79fa644190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7409551e595b4e609349f7360221932c"
I20260807 02:22:23.725690 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a650c557d13433dbedf40b9085f7c10" candidate_uuid: "b323f506ba084d9c84bcdd79fa644190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "026602797d4341d8aa58fcbbd125db2e"
I20260807 02:22:23.725859 12896 raft_consensus.cc:3037] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:23.725867 13187 raft_consensus.cc:3037] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:23.726913 12896 raft_consensus.cc:2454] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b323f506ba084d9c84bcdd79fa644190 in term 1.
I20260807 02:22:23.726917 13187 raft_consensus.cc:2454] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b323f506ba084d9c84bcdd79fa644190 in term 1.
I20260807 02:22:23.727311 12971 leader_election.cc:302] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7409551e595b4e609349f7360221932c, b323f506ba084d9c84bcdd79fa644190; no voters:
I20260807 02:22:23.727522 13270 raft_consensus.cc:2781] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:23.727730 13270 raft_consensus.cc:686] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [term 1 LEADER]: Becoming Leader. State: Replica: b323f506ba084d9c84bcdd79fa644190, State: Running, Role: LEADER
I20260807 02:22:23.727907 13270 consensus_queue.cc:237] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } }
I20260807 02:22:23.729799 12731 catalog_manager.cc:5495] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 reported cstate change: term changed from 0 to 1, leader changed from <none> to b323f506ba084d9c84bcdd79fa644190 (127.4.221.66). New cstate: current_term: 1 leader_uuid: "b323f506ba084d9c84bcdd79fa644190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b323f506ba084d9c84bcdd79fa644190" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 32967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 } health_report { overall_health: UNKNOWN } } }
I20260807 02:22:23.762538 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:23.759309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.4.221.66:32967)
0807 02:22:23.759446 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:23.759450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:23.759466 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:22:23.759558 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:23.759561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:23.759617 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:23.759704 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:22:23.759971 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.760566 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.761401 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:23.761625 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:23.761745 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:23.762075 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:22:23.762146 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:22:23.762226 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:23.762331 (+ 105us) server_negotiation.cc:299] Negotiation successful
0807 02:22:23.762385 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:22:23.769989 12896 raft_consensus.cc:1261] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Refusing update from remote peer b323f506ba084d9c84bcdd79fa644190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:23.770017 13187 raft_consensus.cc:1261] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Refusing update from remote peer b323f506ba084d9c84bcdd79fa644190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:23.770409 13353 consensus_queue.cc:1035] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "026602797d4341d8aa58fcbbd125db2e" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:23.770643 13270 consensus_queue.cc:1035] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7409551e595b4e609349f7360221932c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:23.800702 12734 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:23.800875 12734 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:23.802886 12734 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29c21cb802aa4a0ba568ba4cfa1df249: Sending DeleteTablet for 3 replicas of tablet 1a650c557d13433dbedf40b9085f7c10
I20260807 02:22:23.803507 12875 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a650c557d13433dbedf40b9085f7c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:23 UTC) from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:59766
I20260807 02:22:23.803512 13021 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a650c557d13433dbedf40b9085f7c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:23 UTC) from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:35190
I20260807 02:22:23.803567 13166 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a650c557d13433dbedf40b9085f7c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:23 UTC) from {username='oryx', principal='oryx/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:54928
I20260807 02:22:23.803690 13290 tablet_replica.cc:330] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c: stopping tablet replica
I20260807 02:22:23.803733 13288 tablet_replica.cc:330] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190: stopping tablet replica
I20260807 02:22:23.803807 13290 raft_consensus.cc:2229] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:23.803841 13288 raft_consensus.cc:2229] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:22:23.803906 13290 raft_consensus.cc:2258] T 1a650c557d13433dbedf40b9085f7c10 P 7409551e595b4e609349f7360221932c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:23.804028 13288 raft_consensus.cc:2258] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:23.804214 13289 tablet_replica.cc:330] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e: stopping tablet replica
I20260807 02:22:23.804348 13289 raft_consensus.cc:2229] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:23.804469 13289 raft_consensus.cc:2258] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:23.804746 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 12808
I20260807 02:22:23.806697 13289 ts_tablet_manager.cc:1905] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:23.809684 13289 ts_tablet_manager.cc:1918] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:22:23.809765 13289 log.cc:1196] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/1a650c557d13433dbedf40b9085f7c10
I20260807 02:22:23.810053 13289 ts_tablet_manager.cc:1939] T 1a650c557d13433dbedf40b9085f7c10 P 026602797d4341d8aa58fcbbd125db2e: Deleting consensus metadata
I20260807 02:22:23.810745 12719 catalog_manager.cc:4841] TS 026602797d4341d8aa58fcbbd125db2e (127.4.221.67:44083): tablet 1a650c557d13433dbedf40b9085f7c10 (table test-table [id=c2c575cce7ae42ac8b71c6b479dd4a18]) successfully deleted
I20260807 02:22:23.812325 13288 ts_tablet_manager.cc:1905] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:23.814394 13288 ts_tablet_manager.cc:1918] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260807 02:22:23.814491 13288 log.cc:1196] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/1a650c557d13433dbedf40b9085f7c10
I20260807 02:22:23.814760 13288 ts_tablet_manager.cc:1939] T 1a650c557d13433dbedf40b9085f7c10 P b323f506ba084d9c84bcdd79fa644190: Deleting consensus metadata
I20260807 02:22:23.815444 12720 catalog_manager.cc:4841] TS b323f506ba084d9c84bcdd79fa644190 (127.4.221.66:32967): tablet 1a650c557d13433dbedf40b9085f7c10 (table test-table [id=c2c575cce7ae42ac8b71c6b479dd4a18]) successfully deleted
W20260807 02:22:23.819015 12720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:33021 (error 108)
I20260807 02:22:23.819132 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 12952
I20260807 02:22:23.820542 13363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:23.820043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33021 (local address 127.0.0.1:59784)
0807 02:22:23.820335 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:23.820415 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:33021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":68,"threads_started":1}
W20260807 02:22:23.821192 12720 catalog_manager.cc:4584] TS 7409551e595b4e609349f7360221932c (127.4.221.65:33021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a650c557d13433dbedf40b9085f7c10: Network error: Client connection negotiation failed: client connection to 127.4.221.65:33021: connect: Connection refused (error 111)
I20260807 02:22:23.828238 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 13099
I20260807 02:22:23.837160 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 12702
2026-08-07T02:22:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3498 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:22:23 dist-test-slave-69rg krb5kdc[13366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:22:23 dist-test-slave-69rg krb5kdc[13366](info): set up 2 sockets
Aug 07 02:22:23 dist-test-slave-69rg krb5kdc[13366](info): commencing operation
krb5kdc: starting...
W20260807 02:22:24.909693 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:22:24 dist-test-slave-69rg krb5kdc[13366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:22:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:22:25.048986 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44715
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:44715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:25.167917 13379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:25.168218 13379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:25.168260 13379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:25.172465 13379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:25.172542 13379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:25.172562 13379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:25.172580 13379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:25.172638 13379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:22:25.178421 13379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:44715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:25.179699 13379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:25.180804 13379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:25.189750 13389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:25.189780 13386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:25.189752 13387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:25.190233 13379 server_base.cc:1034] running on GCE node
I20260807 02:22:25.190752 13379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:25.191807 13379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:25.192993 13379 hybrid_clock.cc:648] HybridClock initialized: now 1786069345192973 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-07T02:22:25Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1368 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:22:25 dist-test-slave-69rg krb5kdc[13394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:22:25 dist-test-slave-69rg krb5kdc[13394](info): set up 2 sockets
Aug 07 02:22:25 dist-test-slave-69rg krb5kdc[13394](info): commencing operation
krb5kdc: starting...
W20260807 02:22:26.270256 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:22:26 dist-test-slave-69rg krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:22:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:22:26.413599 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:32833
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:32833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:26.531941 13408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:26.532269 13408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:26.532359 13408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:26.536607 13408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:26.536692 13408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:26.536712 13408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:26.536731 13408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:26.536747 13408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:22:26.542799 13408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:32833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:32833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:26.544075 13408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:26.545197 13408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:26.554729 13417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:26.554767 13414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:26.554734 13415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:26.555095 13408 server_base.cc:1034] running on GCE node
I20260807 02:22:26.555747 13408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:26.557216 13408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:26.558459 13408 hybrid_clock.cc:648] HybridClock initialized: now 1786069346558435 us; error 52 us; skew 500 ppm
Aug 07 02:22:26 dist-test-slave-69rg krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069346, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:26.561658 13408 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:22:26.562805 13408 webserver.cc:460] Webserver started at http://127.4.221.126:36739/ using document root <none> and password file <none>
I20260807 02:22:26.563443 13408 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:26.563545 13408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:26.563794 13408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:26.566138 13408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "f27675e648d74dd8a35886326322dbe6"
format_stamp: "Formatted at 2026-08-07 02:22:26 on dist-test-slave-69rg"
server_key: "1bc621f21cccecd6fd7501d047fdf25d"
server_key_iv: "450c662f9119662d76da66160014f127"
server_key_version: "encryptionkey@0"
I20260807 02:22:26.566767 13408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "f27675e648d74dd8a35886326322dbe6"
format_stamp: "Formatted at 2026-08-07 02:22:26 on dist-test-slave-69rg"
server_key: "1bc621f21cccecd6fd7501d047fdf25d"
server_key_iv: "450c662f9119662d76da66160014f127"
server_key_version: "encryptionkey@0"
I20260807 02:22:26.571094 13408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:22:26.574019 13424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:26.575520 13408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 02:22:26.575707 13408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "f27675e648d74dd8a35886326322dbe6"
format_stamp: "Formatted at 2026-08-07 02:22:26 on dist-test-slave-69rg"
server_key: "1bc621f21cccecd6fd7501d047fdf25d"
server_key_iv: "450c662f9119662d76da66160014f127"
server_key_version: "encryptionkey@0"
I20260807 02:22:26.575858 13408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:26.590699 13408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:26.594295 13408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:26.594530 13408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:26.602721 13408 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:32833
I20260807 02:22:26.602751 13480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:32833 every 8 connection(s)
I20260807 02:22:26.603825 13408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:26.606815 13481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:26.611636 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 13408
I20260807 02:22:26.611801 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:22:26.612106 4981 external_mini_cluster.cc:1428] Setting key 31ec0bd836e6c6fcd75f2bfa6dd7d877
I20260807 02:22:26.613543 13481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6: Bootstrap starting.
I20260807 02:22:26.616137 13481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:26.617085 13481 log.cc:824] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:26.619623 13481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6: No bootstrap required, opened a new log
Aug 07 02:22:26 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:26.624915 13481 raft_consensus.cc:348] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27675e648d74dd8a35886326322dbe6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32833 } }
I20260807 02:22:26.625278 13481 raft_consensus.cc:374] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:26.625367 13481 raft_consensus.cc:729] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f27675e648d74dd8a35886326322dbe6, State: Initialized, Role: FOLLOWER
I20260807 02:22:26.625970 13481 consensus_queue.cc:260] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27675e648d74dd8a35886326322dbe6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32833 } }
I20260807 02:22:26.626166 13481 raft_consensus.cc:388] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:26.626242 13481 raft_consensus.cc:482] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:26.626348 13481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:26.627404 13481 raft_consensus.cc:504] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27675e648d74dd8a35886326322dbe6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32833 } }
I20260807 02:22:26.627804 13481 leader_election.cc:302] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f27675e648d74dd8a35886326322dbe6; no voters:
I20260807 02:22:26.628199 13481 leader_election.cc:288] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:22:26.628293 13486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:26.628486 13486 raft_consensus.cc:686] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 1 LEADER]: Becoming Leader. State: Replica: f27675e648d74dd8a35886326322dbe6, State: Running, Role: LEADER
I20260807 02:22:26.628902 13486 consensus_queue.cc:237] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27675e648d74dd8a35886326322dbe6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32833 } }
I20260807 02:22:26.629097 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:26.613767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.4.221.126:32833)
0807 02:22:26.614307 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:26.614339 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:26.614377 (+ 38us) server_negotiation.cc:407] Connection header received
0807 02:22:26.615051 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:26.615068 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:26.615419 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:26.615719 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:26.616774 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:26.618081 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:26.619335 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:26.619876 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:26.620000 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:26.623083 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:26.623093 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:26.623121 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:26.623163 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:26.625256 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:26.625867 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:26.625873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:26.625879 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:26.626034 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:26.626377 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:26.626380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:26.626384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:26.626890 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:26.627132 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:26.627481 (+ 349us) server_negotiation.cc:299] Negotiation successful
0807 02:22:26.627865 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":155,"threads_started":1}
I20260807 02:22:26.629621 13481 sys_catalog.cc:564] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:26.630786 13487 sys_catalog.cc:455] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f27675e648d74dd8a35886326322dbe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27675e648d74dd8a35886326322dbe6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32833 } } }
I20260807 02:22:26.630918 13487 sys_catalog.cc:458] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:26.631284 13488 sys_catalog.cc:455] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f27675e648d74dd8a35886326322dbe6. Latest consensus state: current_term: 1 leader_uuid: "f27675e648d74dd8a35886326322dbe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27675e648d74dd8a35886326322dbe6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32833 } } }
I20260807 02:22:26.631377 13488 sys_catalog.cc:458] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:26.631558 13490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:26.634817 13490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:26.641065 13490 catalog_manager.cc:1347] Generated new cluster ID: 3c2f24f8498e4a9fb25e0d18d57571b7
I20260807 02:22:26.641172 13490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:26.651260 13490 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:22:26.652325 13490 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:26.657339 13490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6: Generated new TSK 0
I20260807 02:22:26.658195 13490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:22:26.701429 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:32833
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:26.824311 13507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:26.824633 13507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:26.824680 13507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:26.829015 13507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:26.829097 13507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:26.829160 13507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:22:26.835299 13507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:32833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:26.836481 13507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:26.837520 13507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:26.848157 13513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:26.848183 13515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:26.848286 13507 server_base.cc:1034] running on GCE node
W20260807 02:22:26.848475 13512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:26.848907 13507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:26.849604 13507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:26.850840 13507 hybrid_clock.cc:648] HybridClock initialized: now 1786069346850824 us; error 36 us; skew 500 ppm
Aug 07 02:22:26 dist-test-slave-69rg krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069346, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:26.853945 13507 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:22:26.854866 13507 webserver.cc:460] Webserver started at http://127.4.221.65:43469/ using document root <none> and password file <none>
I20260807 02:22:26.855436 13507 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:26.855499 13507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:26.855679 13507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:26.857882 13507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "4f3284f8a46b4078bb1476e9286d6974"
format_stamp: "Formatted at 2026-08-07 02:22:26 on dist-test-slave-69rg"
server_key: "66a6d07ad0b55eacfe9f3f4b5b5a26c8"
server_key_iv: "8036982fa8e85121a28983480bd0e9b5"
server_key_version: "encryptionkey@0"
I20260807 02:22:26.858481 13507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "4f3284f8a46b4078bb1476e9286d6974"
format_stamp: "Formatted at 2026-08-07 02:22:26 on dist-test-slave-69rg"
server_key: "66a6d07ad0b55eacfe9f3f4b5b5a26c8"
server_key_iv: "8036982fa8e85121a28983480bd0e9b5"
server_key_version: "encryptionkey@0"
I20260807 02:22:26.862740 13507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 02:22:26.865535 13522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:26.867086 13507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:22:26.867312 13507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "4f3284f8a46b4078bb1476e9286d6974"
format_stamp: "Formatted at 2026-08-07 02:22:26 on dist-test-slave-69rg"
server_key: "66a6d07ad0b55eacfe9f3f4b5b5a26c8"
server_key_iv: "8036982fa8e85121a28983480bd0e9b5"
server_key_version: "encryptionkey@0"
I20260807 02:22:26.867431 13507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:26.889575 13507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:26.893462 13507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:26.893703 13507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:26.894559 13507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:26.895489 13507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:26.895540 13507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:26.895615 13507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:26.895649 13507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:26.905697 13507 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:38895
I20260807 02:22:26.905721 13637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:38895 every 8 connection(s)
I20260807 02:22:26.906749 13507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:22:26.909745 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 13507
I20260807 02:22:26.909940 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:22:26.910336 4981 external_mini_cluster.cc:1428] Setting key 4c8cfa50fa9f7486d4b5156171700ce2
Aug 07 02:22:26 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069346, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
I20260807 02:22:26.923918 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:26.908798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:46361 (local address 127.4.221.126:32833)
0807 02:22:26.908949 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:26.908954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:26.910044 (+ 1090us) server_negotiation.cc:407] Connection header received
0807 02:22:26.911899 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:26.911903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:26.911954 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:26.912034 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:26.913362 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:26.914056 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:26.915344 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:26.915618 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:26.915711 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:26.918870 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:26.918878 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:26.918885 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:26.918931 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:26.921134 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:26.921772 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:26.921778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:26.921784 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:26.921971 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:26.922410 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:26.922415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:26.922416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:26.922728 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:26.922860 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:26.923549 (+ 689us) server_negotiation.cc:299] Negotiation successful
0807 02:22:26.923725 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 02:22:26.924955 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:26.909167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:46361)
0807 02:22:26.909839 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:26.909916 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:26.911660 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:26.912238 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:26.912245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:26.912657 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:26.913194 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:26.913209 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:26.914230 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:26.914242 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:26.915188 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:26.915199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:26.915787 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:26.915790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:26.915957 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:26.916651 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:26.916668 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:26.918648 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:26.921311 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:26.921317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:26.921333 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:26.921645 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:26.922107 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:26.922110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:26.922111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:26.922290 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:26.922876 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:26.922882 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:26.923220 (+ 338us) client_negotiation.cc:769] Sending connection context
0807 02:22:26.923502 (+ 282us) client_negotiation.cc:240] Negotiation successful
0807 02:22:26.923889 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
I20260807 02:22:26.926457 13638 heartbeater.cc:347] Connected to a master server at 127.4.221.126:32833
I20260807 02:22:26.926735 13638 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:26.927452 13638 heartbeater.cc:511] Master 127.4.221.126:32833 requested a full tablet report, sending...
I20260807 02:22:26.929596 13443 ts_manager.cc:194] Registered new tserver with Master: 4f3284f8a46b4078bb1476e9286d6974 (127.4.221.65:38895)
I20260807 02:22:26.931069 13443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:46361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:22:26.946066 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:32833
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:27.065088 13643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:27.065469 13643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:27.065519 13643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:27.069875 13643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:27.069964 13643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:27.070077 13643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:22:27.076012 13643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:32833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:27.077297 13643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:27.078567 13643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:27.089304 13648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:27.089304 13649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:27.089332 13651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:27.089349 13643 server_base.cc:1034] running on GCE node
I20260807 02:22:27.090204 13643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:27.090915 13643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:27.092123 13643 hybrid_clock.cc:648] HybridClock initialized: now 1786069347092106 us; error 31 us; skew 500 ppm
Aug 07 02:22:27 dist-test-slave-69rg krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069347, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:27.095386 13643 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:22:27.096459 13643 webserver.cc:460] Webserver started at http://127.4.221.66:42379/ using document root <none> and password file <none>
I20260807 02:22:27.097182 13643 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:27.097271 13643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:27.097532 13643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:27.099836 13643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "65719a66bdc94261b3e95762592b4efa"
format_stamp: "Formatted at 2026-08-07 02:22:27 on dist-test-slave-69rg"
server_key: "7be45321b976bc6fce29066cb9a95967"
server_key_iv: "2b1ac36e8174692d66adc56dec41ca4a"
server_key_version: "encryptionkey@0"
I20260807 02:22:27.100457 13643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "65719a66bdc94261b3e95762592b4efa"
format_stamp: "Formatted at 2026-08-07 02:22:27 on dist-test-slave-69rg"
server_key: "7be45321b976bc6fce29066cb9a95967"
server_key_iv: "2b1ac36e8174692d66adc56dec41ca4a"
server_key_version: "encryptionkey@0"
I20260807 02:22:27.104662 13643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260807 02:22:27.107482 13658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:27.108835 13643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:22:27.108994 13643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "65719a66bdc94261b3e95762592b4efa"
format_stamp: "Formatted at 2026-08-07 02:22:27 on dist-test-slave-69rg"
server_key: "7be45321b976bc6fce29066cb9a95967"
server_key_iv: "2b1ac36e8174692d66adc56dec41ca4a"
server_key_version: "encryptionkey@0"
I20260807 02:22:27.109131 13643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:27.159446 13643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:27.162707 13643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:27.162933 13643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:27.163560 13643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:27.164456 13643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:27.164525 13643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:27.164584 13643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:27.164602 13643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:27.174187 13643 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:45821
I20260807 02:22:27.174278 13774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:45821 every 8 connection(s)
I20260807 02:22:27.175213 13643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:22:27.185097 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 13643
I20260807 02:22:27.185233 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
Aug 07 02:22:27 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069347, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:27.185611 4981 external_mini_cluster.cc:1428] Setting key 51ce790b935c9645e4032c469383734d
I20260807 02:22:27.190227 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:27.177108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:48513 (local address 127.4.221.126:32833)
0807 02:22:27.177327 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:27.177333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:27.178250 (+ 917us) server_negotiation.cc:407] Connection header received
0807 02:22:27.179650 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:27.179653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:27.179713 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:27.179787 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:27.181129 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:27.181663 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:27.182912 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:27.183136 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:27.183231 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:27.186281 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:27.186284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:27.186286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:27.186313 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:27.187922 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:27.188517 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:27.188526 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:27.188527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:27.188625 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:27.189004 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:27.189008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:27.189010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:27.189211 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:27.189323 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:27.189911 (+ 588us) server_negotiation.cc:299] Negotiation successful
0807 02:22:27.190056 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260807 02:22:27.191119 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:27.177526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:48513)
0807 02:22:27.178061 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:27.178105 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:27.179386 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:27.179991 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:27.179998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:27.180456 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:27.180986 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:27.181000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:27.181876 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:27.181880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:27.182776 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:27.182786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:27.183315 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:27.183318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:27.183489 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:27.184173 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:27.184189 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:27.186068 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:27.188061 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:27.188067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:27.188085 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:27.188396 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:27.188729 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:27.188733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:27.188735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:27.188896 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:27.189326 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:27.189336 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:27.189625 (+ 289us) client_negotiation.cc:769] Sending connection context
0807 02:22:27.189828 (+ 203us) client_negotiation.cc:240] Negotiation successful
0807 02:22:27.190227 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":132,"threads_started":1}
I20260807 02:22:27.192348 13775 heartbeater.cc:347] Connected to a master server at 127.4.221.126:32833
I20260807 02:22:27.192633 13775 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:27.193295 13775 heartbeater.cc:511] Master 127.4.221.126:32833 requested a full tablet report, sending...
I20260807 02:22:27.194646 13443 ts_manager.cc:194] Registered new tserver with Master: 65719a66bdc94261b3e95762592b4efa (127.4.221.66:45821)
I20260807 02:22:27.195261 13443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:48513
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:22:27.217085 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:32833
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:27.338768 13780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:27.339098 13780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:27.339145 13780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:27.343335 13780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:27.343405 13780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:27.343465 13780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:22:27.349061 13780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:32833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:27.350312 13780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:27.351374 13780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:27.361907 13786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:27.361907 13785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:27.361909 13788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:27.362344 13780 server_base.cc:1034] running on GCE node
I20260807 02:22:27.362824 13780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:27.363507 13780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:27.364697 13780 hybrid_clock.cc:648] HybridClock initialized: now 1786069347364653 us; error 39 us; skew 500 ppm
Aug 07 02:22:27 dist-test-slave-69rg krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069347, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:27.368062 13780 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:22:27.369033 13780 webserver.cc:460] Webserver started at http://127.4.221.67:45079/ using document root <none> and password file <none>
I20260807 02:22:27.369629 13780 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:27.369731 13780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:27.370020 13780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:27.372470 13780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "d75d485ac86c4a35bbaeeb6abc252f39"
format_stamp: "Formatted at 2026-08-07 02:22:27 on dist-test-slave-69rg"
server_key: "e3951923a90d68f8e015bb8b6d6578b2"
server_key_iv: "2e19f56fa2127c47cfe4b8489e0de941"
server_key_version: "encryptionkey@0"
I20260807 02:22:27.373093 13780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "d75d485ac86c4a35bbaeeb6abc252f39"
format_stamp: "Formatted at 2026-08-07 02:22:27 on dist-test-slave-69rg"
server_key: "e3951923a90d68f8e015bb8b6d6578b2"
server_key_iv: "2e19f56fa2127c47cfe4b8489e0de941"
server_key_version: "encryptionkey@0"
I20260807 02:22:27.377278 13780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 02:22:27.380231 13795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:27.381776 13780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:22:27.381989 13780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "d75d485ac86c4a35bbaeeb6abc252f39"
format_stamp: "Formatted at 2026-08-07 02:22:27 on dist-test-slave-69rg"
server_key: "e3951923a90d68f8e015bb8b6d6578b2"
server_key_iv: "2e19f56fa2127c47cfe4b8489e0de941"
server_key_version: "encryptionkey@0"
I20260807 02:22:27.382129 13780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:27.397675 13780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:27.401262 13780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:27.401496 13780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:27.402168 13780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:27.403044 13780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:27.403118 13780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:27.403193 13780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:27.403239 13780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:27.413123 13780 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:46659
I20260807 02:22:27.413178 13910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:46659 every 8 connection(s)
I20260807 02:22:27.414258 13780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:22:27.415684 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 13780
I20260807 02:22:27.415850 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:22:27.416173 4981 external_mini_cluster.cc:1428] Setting key c9bf3309832742d2ca3f91a1474f5298
Aug 07 02:22:27 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069347, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:27.429972 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:27.416430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:39669 (local address 127.4.221.126:32833)
0807 02:22:27.416614 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:27.416619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:27.417734 (+ 1115us) server_negotiation.cc:407] Connection header received
0807 02:22:27.419324 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:27.419327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:27.419374 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:27.419444 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:27.420802 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:27.421416 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:27.422740 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:27.422963 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:27.423047 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:27.426016 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:27.426019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:27.426022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:27.426055 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:27.427651 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:27.428240 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:27.428243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:27.428245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:27.428343 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:27.428677 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:27.428680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:27.428681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:27.428938 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:27.429041 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:27.429619 (+ 578us) server_negotiation.cc:299] Negotiation successful
0807 02:22:27.429768 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 02:22:27.431054 13913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:27.416916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:39669)
0807 02:22:27.417540 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:27.417612 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:27.419110 (+ 1498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:27.419618 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:27.419625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:27.420019 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:27.420656 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:27.420673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:27.421558 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:27.421561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:27.422610 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:27.422620 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:27.423116 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:27.423160 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:27.423332 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:27.423989 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:27.424007 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:27.425763 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:27.427784 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:27.427790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:27.427803 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:27.428084 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:27.428431 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:27.428434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:27.428435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:27.428584 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:27.429039 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:27.429044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:27.429303 (+ 259us) client_negotiation.cc:769] Sending connection context
0807 02:22:27.429555 (+ 252us) client_negotiation.cc:240] Negotiation successful
0807 02:22:27.429965 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":173,"threads_started":1}
I20260807 02:22:27.432315 13911 heartbeater.cc:347] Connected to a master server at 127.4.221.126:32833
I20260807 02:22:27.432567 13911 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:27.433072 13911 heartbeater.cc:511] Master 127.4.221.126:32833 requested a full tablet report, sending...
I20260807 02:22:27.434103 13443 ts_manager.cc:194] Registered new tserver with Master: d75d485ac86c4a35bbaeeb6abc252f39 (127.4.221.67:46659)
I20260807 02:22:27.434696 13443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:39669
I20260807 02:22:27.443077 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.65.keytab.
Entry for principal oryx/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.65.keytab.
WARNING: no policy specified for oryx/127.4.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.66.keytab.
Entry for principal oryx/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.66.keytab.
WARNING: no policy specified for oryx/127.4.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.67.keytab.
Entry for principal oryx/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.67.keytab.
I20260807 02:22:27.533545 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 13507
I20260807 02:22:27.540616 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 13643
I20260807 02:22:27.548309 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 13780
I20260807 02:22:27.555044 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 13408
I20260807 02:22:27.563445 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:32833
--webserver_interface=127.4.221.126
--webserver_port=36739
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:32833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:27.683151 13920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:27.683528 13920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:27.683610 13920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:27.687819 13920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:27.687938 13920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:27.687979 13920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:27.688019 13920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:27.688060 13920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:22:27.693678 13920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:32833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:32833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=36739
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:27.694936 13920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:27.696045 13920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 02:22:27.704995 13920 server_base.cc:1034] running on GCE node
W20260807 02:22:27.704988 13925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:27.704983 13926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:27.705173 13928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:27.705896 13920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:27.707008 13920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:27.708192 13920 hybrid_clock.cc:648] HybridClock initialized: now 1786069347708171 us; error 34 us; skew 500 ppm
Aug 07 02:22:27 dist-test-slave-69rg krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069347, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:27.711193 13920 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:22:27.712174 13920 webserver.cc:460] Webserver started at http://127.4.221.126:36739/ using document root <none> and password file <none>
I20260807 02:22:27.712767 13920 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:27.712886 13920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:27.716883 13920 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260807 02:22:27.719058 13936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:27.720232 13920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 02:22:27.720381 13920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "f27675e648d74dd8a35886326322dbe6"
format_stamp: "Formatted at 2026-08-07 02:22:26 on dist-test-slave-69rg"
server_key: "1bc621f21cccecd6fd7501d047fdf25d"
server_key_iv: "450c662f9119662d76da66160014f127"
server_key_version: "encryptionkey@0"
I20260807 02:22:27.720960 13920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:27.735360 13920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:27.738572 13920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:27.738840 13920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:27.746479 13920 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:32833
I20260807 02:22:27.746506 13992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:32833 every 8 connection(s)
I20260807 02:22:27.747639 13920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:27.751430 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 13920
I20260807 02:22:27.751943 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:38895
--local_ip_for_outbound_sockets=127.4.221.65
--tserver_master_addrs=127.4.221.126:32833
--webserver_port=43469
--webserver_interface=127.4.221.65
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.221.65
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:22:27.752517 13993 sys_catalog.cc:263] Verifying existing consensus state
I20260807 02:22:27.754459 13993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6: Bootstrap starting.
I20260807 02:22:27.779452 13993 log.cc:824] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:27.787961 13993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:22:27.788432 13993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6: Bootstrap complete.
I20260807 02:22:27.793224 13993 raft_consensus.cc:348] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27675e648d74dd8a35886326322dbe6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32833 } }
I20260807 02:22:27.793689 13993 raft_consensus.cc:729] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f27675e648d74dd8a35886326322dbe6, State: Initialized, Role: FOLLOWER
I20260807 02:22:27.794317 13993 consensus_queue.cc:260] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27675e648d74dd8a35886326322dbe6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32833 } }
I20260807 02:22:27.794477 13993 raft_consensus.cc:388] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:27.794571 13993 raft_consensus.cc:482] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:27.794688 13993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 1 FOLLOWER]: Advancing to term 2
I20260807 02:22:27.796694 13993 raft_consensus.cc:504] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27675e648d74dd8a35886326322dbe6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32833 } }
I20260807 02:22:27.797051 13993 leader_election.cc:302] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f27675e648d74dd8a35886326322dbe6; no voters:
I20260807 02:22:27.797436 13993 leader_election.cc:288] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 02:22:27.797534 13997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 02:22:27.797765 13997 raft_consensus.cc:686] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [term 2 LEADER]: Becoming Leader. State: Replica: f27675e648d74dd8a35886326322dbe6, State: Running, Role: LEADER
I20260807 02:22:27.798105 13997 consensus_queue.cc:237] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27675e648d74dd8a35886326322dbe6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32833 } }
I20260807 02:22:27.798885 13993 sys_catalog.cc:564] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:27.799130 13999 sys_catalog.cc:455] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f27675e648d74dd8a35886326322dbe6. Latest consensus state: current_term: 2 leader_uuid: "f27675e648d74dd8a35886326322dbe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27675e648d74dd8a35886326322dbe6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32833 } } }
I20260807 02:22:27.799283 13999 sys_catalog.cc:458] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:27.799522 13998 sys_catalog.cc:455] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f27675e648d74dd8a35886326322dbe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f27675e648d74dd8a35886326322dbe6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 32833 } } }
I20260807 02:22:27.799626 13998 sys_catalog.cc:458] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:27.800056 14001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:27.803635 14001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:27.804347 14001 catalog_manager.cc:1259] Loaded cluster ID: 3c2f24f8498e4a9fb25e0d18d57571b7
I20260807 02:22:27.804430 14001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:27.806285 14001 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:27.806887 14001 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f27675e648d74dd8a35886326322dbe6: Loaded TSK: 0
I20260807 02:22:27.807878 14001 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 02:22:27.902455 13995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:27.902765 13995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:27.902841 13995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:27.907200 13995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:27.907320 13995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:27.907420 13995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:22:27.913399 13995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.65.keytab
--principal=oryx/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:38895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=43469
--tserver_master_addrs=127.4.221.126:32833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:27.914609 13995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:27.915689 13995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:27.926102 14019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:27.926095 14022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:27.926322 13995 server_base.cc:1034] running on GCE node
W20260807 02:22:27.926139 14020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:27.926955 13995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:27.927719 13995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:27.928915 13995 hybrid_clock.cc:648] HybridClock initialized: now 1786069347928900 us; error 29 us; skew 500 ppm
Aug 07 02:22:27 dist-test-slave-69rg krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069347, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:27.932215 13995 init.cc:377] Logged in from keytab as oryx/127.4.221.65@KRBTEST.COM (short username oryx)
I20260807 02:22:27.933372 13995 webserver.cc:460] Webserver started at http://127.4.221.65:43469/ using document root <none> and password file <none>
I20260807 02:22:27.934152 13995 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:27.934270 13995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:27.939694 13995 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:22:27.942495 14029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:27.944114 13995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:22:27.944259 13995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "4f3284f8a46b4078bb1476e9286d6974"
format_stamp: "Formatted at 2026-08-07 02:22:26 on dist-test-slave-69rg"
server_key: "66a6d07ad0b55eacfe9f3f4b5b5a26c8"
server_key_iv: "8036982fa8e85121a28983480bd0e9b5"
server_key_version: "encryptionkey@0"
I20260807 02:22:27.945040 13995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:27.969888 13995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:27.973673 13995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:27.973925 13995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:27.974674 13995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:27.975807 13995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:27.975871 13995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:27.975925 13995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:27.975948 13995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:27.986294 13995 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:38895
I20260807 02:22:27.986351 14145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:38895 every 8 connection(s)
I20260807 02:22:27.987355 13995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:22:27.991658 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 13995
I20260807 02:22:27.992198 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:45821
--local_ip_for_outbound_sockets=127.4.221.66
--tserver_master_addrs=127.4.221.126:32833
--webserver_port=42379
--webserver_interface=127.4.221.66
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.221.66
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:22:27 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:27 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:28.002269 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:27.989617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:41985)
0807 02:22:27.990105 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:27.990147 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:27.991574 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:27.992958 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:27.992965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:27.993375 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:27.993935 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:27.993960 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:27.995391 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:27.995394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:27.996288 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:27.996298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:27.997476 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:27.997480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:27.997614 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:27.998367 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:27.998385 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:28.001311 (+ 2926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":173,"threads_started":1}
W20260807 02:22:28.002866 14146 heartbeater.cc:644] Failed to heartbeat to 127.4.221.126:32833 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.221.126:32833: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
W20260807 02:22:28.003438 14149 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:27.989470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:41985 (local address 127.4.221.126:32833)
0807 02:22:27.989982 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:27.990003 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:27.990265 (+ 262us) server_negotiation.cc:407] Connection header received
0807 02:22:27.992064 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:27.992082 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:27.992516 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:27.992872 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:27.994173 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:27.995210 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:27.996438 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:27.997293 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:27.997516 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.001697 (+ 4181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:41985: BlockingRecv error: recv got EOF from 127.4.221.65:41985 (error 108)
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":108,"threads_started":1}
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:28.009956 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:28.003446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:35031)
0807 02:22:28.003927 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:28.003942 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:28.004069 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:28.004367 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:28.004371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:28.004563 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:28.004669 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.004674 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.005463 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.005468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.006131 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.006141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.006814 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.006817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.006917 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.007593 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:28.007606 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:28.009756 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394}
W20260807 02:22:28.010069 14152 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:28.003523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:35031 (local address 127.4.221.126:32833)
0807 02:22:28.003875 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:28.003892 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:28.004017 (+ 125us) server_negotiation.cc:407] Connection header received
0807 02:22:28.004182 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:28.004185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:28.004247 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:28.004349 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:28.004791 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.005343 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.006270 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.006641 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.006795 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.009927 (+ 3132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:35031: BlockingRecv error: recv got EOF from 127.4.221.65:35031 (error 108)
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":157,"threads_started":1}
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:28.016499 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:28.010546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:57841)
0807 02:22:28.010717 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:28.010733 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:28.010819 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:28.011090 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:28.011096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:28.011286 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:28.011415 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.011425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.012268 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.012272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.013165 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.013175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.013623 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.013627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.013746 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.014320 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:28.014336 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:28.016342 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
W20260807 02:22:28.016760 14152 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:28.010667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:57841 (local address 127.4.221.126:32833)
0807 02:22:28.010798 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:28.010801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:28.010813 (+ 12us) server_negotiation.cc:407] Connection header received
0807 02:22:28.010929 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:28.010931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:28.010972 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:28.011036 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:28.011597 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.012106 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.013291 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.013509 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.013593 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.016489 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:57841: BlockingRecv error: recv got EOF from 127.4.221.65:57841 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260807 02:22:28.016763 14146 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 02:22:28.118228 14151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:28.118585 14151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:28.118681 14151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:28.123299 14151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:28.123409 14151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:28.123507 14151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:22:28.129554 14151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.66.keytab
--principal=oryx/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:45821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=42379
--tserver_master_addrs=127.4.221.126:32833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:28.130879 14151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:28.132026 14151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:28.142478 14157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:28.142458 14160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:28.142462 14158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:28.142516 14151 server_base.cc:1034] running on GCE node
I20260807 02:22:28.143129 14151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:28.143831 14151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:28.145025 14151 hybrid_clock.cc:648] HybridClock initialized: now 1786069348144996 us; error 39 us; skew 500 ppm
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069348, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:28.148041 14151 init.cc:377] Logged in from keytab as oryx/127.4.221.66@KRBTEST.COM (short username oryx)
I20260807 02:22:28.149070 14151 webserver.cc:460] Webserver started at http://127.4.221.66:42379/ using document root <none> and password file <none>
I20260807 02:22:28.149712 14151 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:28.149855 14151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:28.154089 14151 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:22:28.156397 14167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:28.157667 14151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:22:28.157883 14151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "65719a66bdc94261b3e95762592b4efa"
format_stamp: "Formatted at 2026-08-07 02:22:27 on dist-test-slave-69rg"
server_key: "7be45321b976bc6fce29066cb9a95967"
server_key_iv: "2b1ac36e8174692d66adc56dec41ca4a"
server_key_version: "encryptionkey@0"
I20260807 02:22:28.158569 14151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:28.184940 14151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:28.188452 14151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:28.188699 14151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:28.189399 14151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:28.190333 14151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:28.190411 14151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:28.190487 14151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:28.190541 14151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:28.200395 14151 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:45821
I20260807 02:22:28.200421 14281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:45821 every 8 connection(s)
I20260807 02:22:28.201623 14151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:22:28.211058 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 14151
I20260807 02:22:28.211634 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:46659
--local_ip_for_outbound_sockets=127.4.221.67
--tserver_master_addrs=127.4.221.126:32833
--webserver_port=45079
--webserver_interface=127.4.221.67
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.221.67
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 02:22:28.213994 14152 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:28.203666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:33403 (local address 127.4.221.126:32833)
0807 02:22:28.203825 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:28.203829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:28.204770 (+ 941us) server_negotiation.cc:407] Connection header received
0807 02:22:28.206213 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:28.206217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:28.206287 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:28.206367 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:28.207598 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.208176 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.209369 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.209614 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.209741 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.213873 (+ 4132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:33403: BlockingRecv error: recv got EOF from 127.4.221.66:33403 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 02:22:28.214872 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:28.204083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:33403)
0807 02:22:28.204580 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:28.204627 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:28.205983 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:28.206547 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:28.206554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:28.206934 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:28.207437 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.207450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.208324 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.208327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.209229 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.209240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.209792 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.209796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.210003 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.210596 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:28.210613 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:28.213641 (+ 3028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":137,"threads_started":1}
W20260807 02:22:28.215448 14282 heartbeater.cc:644] Failed to heartbeat to 127.4.221.126:32833 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.221.126:32833: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:28.222738 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:28.215981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:55231)
0807 02:22:28.216189 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:28.216206 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:28.216318 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:28.216595 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:28.216601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:28.216829 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:28.216998 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.217009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.217853 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.217858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.218967 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.218982 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.219511 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.219517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.219656 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.220182 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:28.220206 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:28.222564 (+ 2358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118}
W20260807 02:22:28.222908 14152 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:28.216076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:55231 (local address 127.4.221.126:32833)
0807 02:22:28.216200 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:28.216203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:28.216280 (+ 77us) server_negotiation.cc:407] Connection header received
0807 02:22:28.216419 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:28.216421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:28.216472 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:28.216541 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:28.217146 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.217703 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.219145 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.219378 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.219465 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.222790 (+ 3325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:55231: BlockingRecv error: recv got EOF from 127.4.221.66:55231 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:28.229264 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:28.223548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:38579)
0807 02:22:28.223679 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:28.223697 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:28.223803 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:28.224081 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:28.224084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:28.224258 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:28.224361 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.224367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.225243 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.225248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.225779 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.225787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.226432 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.226434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.226496 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.227039 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:28.227051 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:28.229115 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260807 02:22:28.229394 14152 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:28.223621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:38579 (local address 127.4.221.126:32833)
0807 02:22:28.223750 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:28.223755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:28.223770 (+ 15us) server_negotiation.cc:407] Connection header received
0807 02:22:28.223923 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:28.223926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:28.223980 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:28.224065 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:28.224476 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.225127 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.225942 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.226313 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.226425 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.229270 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:38579: BlockingRecv error: recv got EOF from 127.4.221.66:38579 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260807 02:22:28.229696 14282 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 02:22:28.333966 14285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:28.334303 14285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:28.334380 14285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:28.338562 14285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:28.338665 14285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:28.338759 14285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:22:28.344239 14285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/krb5kdc/oryx_127.4.221.67.keytab
--principal=oryx/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:46659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=45079
--tserver_master_addrs=127.4.221.126:32833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:28.345383 14285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:28.346462 14285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:28.356487 14290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:28.356490 14294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:28.356490 14291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:28.356750 14285 server_base.cc:1034] running on GCE node
I20260807 02:22:28.357288 14285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:28.358130 14285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:28.359326 14285 hybrid_clock.cc:648] HybridClock initialized: now 1786069348359312 us; error 26 us; skew 500 ppm
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069348, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:28.362206 14285 init.cc:377] Logged in from keytab as oryx/127.4.221.67@KRBTEST.COM (short username oryx)
I20260807 02:22:28.363138 14285 webserver.cc:460] Webserver started at http://127.4.221.67:45079/ using document root <none> and password file <none>
I20260807 02:22:28.363758 14285 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:28.363883 14285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:28.367937 14285 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:22:28.370270 14301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:28.371590 14285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:22:28.371744 14285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "d75d485ac86c4a35bbaeeb6abc252f39"
format_stamp: "Formatted at 2026-08-07 02:22:27 on dist-test-slave-69rg"
server_key: "e3951923a90d68f8e015bb8b6d6578b2"
server_key_iv: "2e19f56fa2127c47cfe4b8489e0de941"
server_key_version: "encryptionkey@0"
I20260807 02:22:28.372318 14285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:28.391346 14285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:28.394546 14285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:28.394795 14285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:28.395442 14285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:28.396301 14285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:28.396377 14285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:28.396450 14285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:28.396497 14285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:28.406213 14285 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:46659
I20260807 02:22:28.406246 14420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:46659 every 8 connection(s)
I20260807 02:22:28.407308 14285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:22:28.411022 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 14285
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 02:22:28.419624 14152 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:28.409410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:44625 (local address 127.4.221.126:32833)
0807 02:22:28.409585 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:28.409589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:28.410523 (+ 934us) server_negotiation.cc:407] Connection header received
0807 02:22:28.412376 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:28.412379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:28.412438 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:28.412549 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:28.413772 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.414332 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.415482 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.415817 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.415922 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.419487 (+ 3565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:44625: BlockingRecv error: recv got EOF from 127.4.221.67:44625 (error 108)
Metrics: {"server-negotiator.queue_time_us":73,"spinlock_wait_cycles":29184}
I20260807 02:22:28.420303 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:28.409760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:44625)
0807 02:22:28.410337 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:28.410384 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:28.412120 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:28.412747 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:28.412755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:28.413126 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:28.413619 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.413633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.414468 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.414472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.415355 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.415364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.415998 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.416004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.416212 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.416769 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:28.416793 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:28.419379 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":139,"threads_started":1}
W20260807 02:22:28.420851 14421 heartbeater.cc:644] Failed to heartbeat to 127.4.221.126:32833 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.221.126:32833: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
I20260807 02:22:28.421507 14149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:28.412266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.4.221.126:32833)
0807 02:22:28.412405 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:28.412408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:28.412419 (+ 11us) server_negotiation.cc:407] Connection header received
0807 02:22:28.412452 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:28.412454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:28.412488 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:28.412589 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:28.413434 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.414037 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.414923 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.415144 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.415262 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.417058 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:28.417063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:28.417077 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:28.417102 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:28.419024 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:28.419503 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:28.419509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:28.419513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:28.419641 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:28.420003 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:28.420007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:28.420009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:28.420359 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:28.420568 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:28.421059 (+ 491us) server_negotiation.cc:299] Negotiation successful
0807 02:22:28.421361 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"spinlock_wait_cycles":896}
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:28.427444 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:28.421363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:55225)
0807 02:22:28.421548 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:28.421564 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:28.421645 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:28.421911 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:28.421913 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:28.422076 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:28.422187 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.422193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.423070 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.423073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.423890 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.423899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.424417 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.424420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.424518 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.425142 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:28.425185 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:28.427280 (+ 2095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108}
W20260807 02:22:28.427558 14152 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:28.421446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:55225 (local address 127.4.221.126:32833)
0807 02:22:28.421583 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:28.421590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:28.421607 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:22:28.421751 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:28.421754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:28.421797 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:28.421891 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:28.422294 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.422967 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.424011 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.424291 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.424394 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.427440 (+ 3046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:55225: BlockingRecv error: recv got EOF from 127.4.221.67:55225 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:28 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:28.433085 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:28.428116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:43597)
0807 02:22:28.428252 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:28.428264 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:28.428348 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:28.428605 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:28.428607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:28.428758 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:28.428841 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.428846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.429577 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.429580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.430186 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:28.430195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.430616 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.430619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:28.430685 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.431140 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:28.431151 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:28.432923 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260807 02:22:28.433169 14152 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:28.428228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:43597 (local address 127.4.221.126:32833)
0807 02:22:28.428407 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:28.428411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:28.428423 (+ 12us) server_negotiation.cc:407] Connection header received
0807 02:22:28.428464 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:28.428466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:28.428504 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:28.428572 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:28.428944 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.429481 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.430311 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:28.430519 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:28.430600 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:28.433049 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:43597: BlockingRecv error: recv got EOF from 127.4.221.67:43597 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260807 02:22:28.433384 14421 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 07 02:22:29 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:29 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:29.023765 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:29.017552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:49357)
0807 02:22:29.018044 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:29.018066 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:29.018187 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:29.018499 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:29.018502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:29.018782 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:29.018913 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:29.018922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:29.019903 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:29.019906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:29.020482 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:29.020491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:29.020964 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:29.020966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:29.021027 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:29.021594 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:29.021615 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:29.023621 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":92,"threads_started":1}
W20260807 02:22:29.024062 14426 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:29.017722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:49357 (local address 127.4.221.126:32833)
0807 02:22:29.018057 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:29.018062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:29.018188 (+ 126us) server_negotiation.cc:407] Connection header received
0807 02:22:29.018316 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:29.018319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:29.018367 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:29.018443 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:29.019054 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:29.019795 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:29.020622 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:29.020861 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:29.020950 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:29.023785 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:49357: BlockingRecv error: recv got EOF from 127.4.221.65:49357 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":123,"threads_started":1}
Aug 07 02:22:29 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:29 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:29.237021 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:29.230475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:47607)
0807 02:22:29.230837 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:29.230853 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:29.230984 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:29.231322 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:29.231325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:29.231581 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:29.231752 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:29.231759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:29.232563 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:29.232566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:29.233245 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:29.233253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:29.233709 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:29.233713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:29.233921 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:29.234604 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:29.234620 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:29.236859 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":159,"threads_started":1}
W20260807 02:22:29.237237 14426 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:29.230573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:47607 (local address 127.4.221.126:32833)
0807 02:22:29.230735 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:29.230739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:29.230936 (+ 197us) server_negotiation.cc:407] Connection header received
0807 02:22:29.231073 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:29.231075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:29.231127 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:29.231246 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:29.231871 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:29.232455 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:29.233349 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:29.233569 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:29.233648 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:29.237031 (+ 3383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:47607: BlockingRecv error: recv got EOF from 127.4.221.66:47607 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 07 02:22:29 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:29 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:29.441299 14428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:29.434278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:46405)
0807 02:22:29.434690 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:29.434712 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:29.434880 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:29.435212 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:29.435215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:29.435485 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:29.435607 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:29.435613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:29.436469 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:29.436474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:29.437103 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:29.437111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:29.437607 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:29.437610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:29.437705 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:29.438469 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:29.438486 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:29.441051 (+ 2565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":138,"threads_started":1}
W20260807 02:22:29.441403 14426 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:29.434318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:46405 (local address 127.4.221.126:32833)
0807 02:22:29.434548 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:29.434553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:29.434808 (+ 255us) server_negotiation.cc:407] Connection header received
0807 02:22:29.435019 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:29.435022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:29.435090 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:29.435184 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:29.435738 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:29.436327 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:29.437230 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:29.437481 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:29.437600 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:29.441230 (+ 3630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:46405: BlockingRecv error: recv got EOF from 127.4.221.67:46405 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 07 02:22:30 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:30 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:30.032289 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:30.024981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:58505)
0807 02:22:30.025366 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:30.025393 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:30.025529 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:30.026005 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:30.026008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:30.026277 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:30.026461 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:30.026470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:30.027801 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:30.027805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:30.028517 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:30.028529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:30.029001 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:30.029004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:30.029076 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:30.029697 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:30.029710 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:30.032091 (+ 2381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
W20260807 02:22:30.032804 14430 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:30.025148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:58505 (local address 127.4.221.126:32833)
0807 02:22:30.025533 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:30.025538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:30.025556 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:22:30.025752 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:30.025756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:30.025843 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:30.025943 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:30.026638 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:30.027529 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:30.028643 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:30.028892 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:30.028978 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:30.032637 (+ 3659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:58505: BlockingRecv error: recv got EOF from 127.4.221.65:58505 (error 108)
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":105,"threads_started":1}
Aug 07 02:22:30 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:30 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:30.248713 14431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:30.238151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:49787)
0807 02:22:30.238651 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:30.238666 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:30.238850 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:30.239213 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:30.239218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:30.239506 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:30.239642 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:30.239649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:30.240680 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:30.240687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:30.241383 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:30.241393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:30.242007 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:30.242011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:30.242122 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:30.246087 (+ 3965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:30.246107 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:30.248468 (+ 2361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":127,"threads_started":1}
W20260807 02:22:30.248802 14430 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:30.238261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:49787 (local address 127.4.221.126:32833)
0807 02:22:30.238466 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:30.238472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:30.238767 (+ 295us) server_negotiation.cc:407] Connection header received
0807 02:22:30.238991 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:30.238996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:30.239078 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:30.239181 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:30.239790 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:30.240544 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:30.241530 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:30.241803 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:30.241999 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:30.248655 (+ 6656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:49787: BlockingRecv error: recv got EOF from 127.4.221.66:49787 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 07 02:22:30 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:30 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:30.449585 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:30.442694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:53517)
0807 02:22:30.443087 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:30.443106 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:30.443267 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:30.443651 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:30.443654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:30.443967 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:30.444121 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:30.444129 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:30.444978 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:30.444984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:30.445590 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:30.445598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:30.446154 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:30.446158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:30.446245 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:30.446884 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:30.446898 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:30.449328 (+ 2430us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":123,"threads_started":1}
W20260807 02:22:30.449652 14430 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:30.442869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:53517 (local address 127.4.221.126:32833)
0807 02:22:30.443036 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:30.443042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:30.443233 (+ 191us) server_negotiation.cc:407] Connection header received
0807 02:22:30.443441 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:30.443445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:30.443517 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:30.443617 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:30.444253 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:30.444859 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:30.445714 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:30.446031 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:30.446150 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:30.449499 (+ 3349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:53517: BlockingRecv error: recv got EOF from 127.4.221.67:53517 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 07 02:22:31 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:31 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:31.040481 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:31.033355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:37821)
0807 02:22:31.033748 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:31.033777 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:31.033952 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:31.034272 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:31.034276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:31.034551 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:31.034689 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:31.034697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:31.035640 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:31.035643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:31.036249 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:31.036258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:31.036826 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:31.036829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:31.036957 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:31.037730 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:31.037743 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:31.040277 (+ 2534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
W20260807 02:22:31.040689 14434 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:31.033559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:37821 (local address 127.4.221.126:32833)
0807 02:22:31.033947 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:31.033952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:31.033968 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:22:31.034037 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:31.034040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:31.034094 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:31.034188 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:31.034839 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:31.035506 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:31.036373 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:31.036670 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:31.036776 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:31.040517 (+ 3741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:37821: BlockingRecv error: recv got EOF from 127.4.221.65:37821 (error 108)
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
Aug 07 02:22:31 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:31 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:31.257351 14435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:31.250075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:38529)
0807 02:22:31.250549 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:31.250574 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:31.250747 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:31.251226 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:31.251229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:31.251618 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:31.251765 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:31.251772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:31.252926 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:31.252930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:31.253561 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:31.253569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:31.254108 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:31.254111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:31.254189 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:31.254776 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:31.254789 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:31.257184 (+ 2395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":126,"threads_started":1}
W20260807 02:22:31.257638 14434 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:31.250171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:38529 (local address 127.4.221.126:32833)
0807 02:22:31.250552 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:31.250583 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:31.250691 (+ 108us) server_negotiation.cc:407] Connection header received
0807 02:22:31.250951 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:31.250955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:31.251036 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:31.251144 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:31.251962 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:31.252788 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:31.253716 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:31.253982 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:31.254078 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:31.257380 (+ 3302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:38529: BlockingRecv error: recv got EOF from 127.4.221.66:38529 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 07 02:22:31 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:31 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:31.456071 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:31.450619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:47123)
0807 02:22:31.450912 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:31.450935 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:31.451043 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:31.451271 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:31.451273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:31.451474 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:31.451585 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:31.451591 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:31.452308 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:31.452310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:31.452889 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:31.452896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:31.453301 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:31.453303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:31.453367 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:31.453930 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:31.453942 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:31.455940 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":106,"threads_started":1}
W20260807 02:22:31.456239 14434 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:31.450687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:47123 (local address 127.4.221.126:32833)
0807 02:22:31.450869 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:31.450875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:31.451015 (+ 140us) server_negotiation.cc:407] Connection header received
0807 02:22:31.451113 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:31.451115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:31.451164 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:31.451231 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:31.451687 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:31.452216 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:31.452993 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:31.453204 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:31.453283 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:31.456107 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:47123: BlockingRecv error: recv got EOF from 127.4.221.67:47123 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 07 02:22:32 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:32 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:32.047088 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:32.041586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:44097)
0807 02:22:32.041916 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:32.041929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:32.042035 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:32.042276 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:32.042278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:32.042498 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:32.042615 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:32.042621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:32.043377 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:32.043380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:32.043946 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:32.043953 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:32.044385 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:32.044388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:32.044455 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:32.044982 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:32.044993 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:32.046956 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":111,"threads_started":1}
W20260807 02:22:32.047256 14438 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:32.041624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:44097 (local address 127.4.221.126:32833)
0807 02:22:32.041919 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:32.041923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:32.041980 (+ 57us) server_negotiation.cc:407] Connection header received
0807 02:22:32.042115 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:32.042117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:32.042161 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:32.042232 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:32.042719 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:32.043268 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:32.044045 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:32.044269 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:32.044345 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:32.047105 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:44097: BlockingRecv error: recv got EOF from 127.4.221.65:44097 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":102,"threads_started":1}
Aug 07 02:22:32 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:32 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:32.265923 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:32.258706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:56047)
0807 02:22:32.259080 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:32.259097 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:32.259264 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:32.259644 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:32.259648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:32.260001 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:32.260165 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:32.260173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:32.261070 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:32.261073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:32.261682 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:32.261690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:32.262266 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:32.262269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:32.262358 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:32.263207 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:32.263226 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:32.265691 (+ 2465us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":119,"threads_started":1}
W20260807 02:22:32.266119 14438 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:32.258790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:56047 (local address 127.4.221.126:32833)
0807 02:22:32.258991 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:32.258996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:32.259170 (+ 174us) server_negotiation.cc:407] Connection header received
0807 02:22:32.259377 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:32.259380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:32.259463 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:32.259559 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:32.260288 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:32.260934 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:32.261798 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:32.262130 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:32.262216 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:32.265941 (+ 3725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:56047: BlockingRecv error: recv got EOF from 127.4.221.66:56047 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 07 02:22:32 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:32 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:32.463966 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:32.457154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:50381)
0807 02:22:32.457618 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:32.457631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:32.457800 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:32.458188 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:32.458190 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:32.458483 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:32.458619 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:32.458625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:32.459469 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:32.459472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:32.460082 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:32.460090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:32.460535 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:32.460538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:32.460611 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:32.461267 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:32.461284 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:32.463803 (+ 2519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":137,"threads_started":1}
W20260807 02:22:32.464170 14438 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:32.457242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:50381 (local address 127.4.221.126:32833)
0807 02:22:32.457407 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:32.457412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:32.457750 (+ 338us) server_negotiation.cc:407] Connection header received
0807 02:22:32.457953 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:32.457957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:32.458026 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:32.458148 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:32.458731 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:32.459321 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:32.460182 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:32.460406 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:32.460523 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:32.463988 (+ 3465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:50381: BlockingRecv error: recv got EOF from 127.4.221.67:50381 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 07 02:22:33 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:33 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:33.055184 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:33.048204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:56531)
0807 02:22:33.048602 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:33.048626 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:33.048738 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:33.049054 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:33.049057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:33.049320 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:33.049488 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:33.049495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:33.050517 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:33.050520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:33.051330 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:33.051343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:33.051840 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:33.051843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:33.051945 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:33.052651 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:33.052665 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:33.055024 (+ 2359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":177,"threads_started":1}
W20260807 02:22:33.055522 14443 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:33.048302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:56531 (local address 127.4.221.126:32833)
0807 02:22:33.048603 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:33.048607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:33.048702 (+ 95us) server_negotiation.cc:407] Connection header received
0807 02:22:33.048813 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:33.048815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:33.048864 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:33.048967 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:33.049625 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:33.050374 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:33.051454 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:33.051706 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:33.051794 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:33.055223 (+ 3429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:56531: BlockingRecv error: recv got EOF from 127.4.221.65:56531 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":122,"threads_started":1}
Aug 07 02:22:33 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:33 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:33.274489 14444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:33.267261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:58657)
0807 02:22:33.267826 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:33.267842 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:33.268014 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:33.268354 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:33.268358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:33.268653 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:33.268786 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:33.268794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:33.269746 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:33.269752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:33.270568 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:33.270578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:33.271134 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:33.271138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:33.271232 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:33.271871 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:33.271885 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:33.274268 (+ 2383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":190,"threads_started":1}
W20260807 02:22:33.274649 14443 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:33.267521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:58657 (local address 127.4.221.126:32833)
0807 02:22:33.267713 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:33.267719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:33.267943 (+ 224us) server_negotiation.cc:407] Connection header received
0807 02:22:33.268146 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:33.268150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:33.268221 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:33.268326 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:33.268935 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:33.269613 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:33.270714 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:33.270994 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:33.271132 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:33.274504 (+ 3372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:58657: BlockingRecv error: recv got EOF from 127.4.221.66:58657 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 07 02:22:33 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:33 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:33.472410 14445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:33.465256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:44595)
0807 02:22:33.465624 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:33.465639 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:33.465765 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:33.466162 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:33.466165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:33.466414 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:33.466569 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:33.466576 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:33.467561 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:33.467564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:33.468213 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:33.468223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:33.468799 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:33.468803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:33.468893 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:33.469706 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:33.469720 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:33.472225 (+ 2505us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":123,"threads_started":1}
W20260807 02:22:33.472579 14443 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:33.465292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:44595 (local address 127.4.221.126:32833)
0807 02:22:33.465432 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:33.465437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:33.465751 (+ 314us) server_negotiation.cc:407] Connection header received
0807 02:22:33.465918 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:33.465921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:33.465986 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:33.466072 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:33.466725 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:33.467389 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:33.468376 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:33.468640 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:33.468738 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:33.472417 (+ 3679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:44595: BlockingRecv error: recv got EOF from 127.4.221.67:44595 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 07 02:22:34 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:34 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:34.064216 14446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:34.056201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:59369)
0807 02:22:34.056564 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:34.056579 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:34.056704 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:34.056997 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:34.057000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:34.057238 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:34.057363 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:34.057369 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:34.058297 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:34.058300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:34.058904 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:34.058915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:34.059655 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:34.059659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:34.059794 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:34.060760 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:34.060777 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:34.063992 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":96,"threads_started":1}
W20260807 02:22:34.064432 14447 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:34.056252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:59369 (local address 127.4.221.126:32833)
0807 02:22:34.056625 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:34.056629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:34.056643 (+ 14us) server_negotiation.cc:407] Connection header received
0807 02:22:34.056802 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:34.056804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:34.056857 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:34.056955 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:34.057513 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:34.058175 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:34.059079 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:34.059479 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:34.059622 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:34.064237 (+ 4615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:59369: BlockingRecv error: recv got EOF from 127.4.221.65:59369 (error 108)
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":129,"threads_started":1}
Aug 07 02:22:34 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:34 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:34.283177 14448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:34.275691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:51131)
0807 02:22:34.276111 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:34.276132 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:34.276329 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:34.276730 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:34.276735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:34.277078 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:34.277215 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:34.277223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:34.278363 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:34.278371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:34.279079 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:34.279090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:34.279689 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:34.279694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:34.279805 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:34.280496 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:34.280512 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:34.283008 (+ 2496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":138,"threads_started":1}
W20260807 02:22:34.283547 14447 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:34.275750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:51131 (local address 127.4.221.126:32833)
0807 02:22:34.275961 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:34.275968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:34.276242 (+ 274us) server_negotiation.cc:407] Connection header received
0807 02:22:34.276501 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:34.276507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:34.276585 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:34.276697 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:34.277357 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:34.278231 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:34.279232 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:34.279537 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:34.279698 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:34.283224 (+ 3526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:51131: BlockingRecv error: recv got EOF from 127.4.221.66:51131 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 07 02:22:34 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:34 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:34.480049 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:34.473484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:59523)
0807 02:22:34.473916 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:34.473934 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:34.474070 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:34.474395 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:34.474398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:34.474662 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:34.474796 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:34.474803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:34.475714 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:34.475718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:34.476450 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:34.476463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:34.476944 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:34.476948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:34.477018 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:34.477623 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:34.477637 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:34.479903 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":122,"threads_started":1}
W20260807 02:22:34.480317 14447 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:34.473618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:59523 (local address 127.4.221.126:32833)
0807 02:22:34.473892 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:34.473896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:34.474022 (+ 126us) server_negotiation.cc:407] Connection header received
0807 02:22:34.474189 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:34.474192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:34.474250 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:34.474323 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:34.474947 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:34.475587 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:34.476584 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:34.476817 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:34.476917 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:34.480071 (+ 3154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:59523: BlockingRecv error: recv got EOF from 127.4.221.67:59523 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 07 02:22:35 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:35 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:35.072786 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:35.065476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:36309)
0807 02:22:35.066032 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:35.066051 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:35.066220 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:35.066666 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:35.066670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:35.067010 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:35.067141 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:35.067149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:35.068034 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:35.068036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:35.068628 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:35.068636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:35.069144 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:35.069147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:35.069220 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:35.069922 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:35.069936 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:35.072609 (+ 2673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":144,"threads_started":1}
W20260807 02:22:35.073122 14450 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:35.065529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:36309 (local address 127.4.221.126:32833)
0807 02:22:35.065983 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:35.065990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:35.066180 (+ 190us) server_negotiation.cc:407] Connection header received
0807 02:22:35.066377 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:35.066383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:35.066525 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:35.066611 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:35.067269 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:35.067930 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:35.068758 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:35.069028 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:35.069127 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:35.072894 (+ 3767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:36309: BlockingRecv error: recv got EOF from 127.4.221.65:36309 (error 108)
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":174,"threads_started":1}
Aug 07 02:22:35 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:35 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:35.296882 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:35.284394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:38721)
0807 02:22:35.284884 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:35.284900 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:35.285041 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:35.285356 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:35.285358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:35.285649 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:35.285777 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:35.285783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:35.286933 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:35.286940 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:35.287909 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:35.287924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:35.288537 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:35.288540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:35.288634 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:35.294380 (+ 5746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:35.294397 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:35.296724 (+ 2327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":169,"threads_started":1}
W20260807 02:22:35.297140 14450 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:35.284533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:38721 (local address 127.4.221.126:32833)
0807 02:22:35.284770 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:35.284775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:35.285027 (+ 252us) server_negotiation.cc:407] Connection header received
0807 02:22:35.285140 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:35.285143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:35.285210 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:35.285313 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:35.285980 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:35.286692 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:35.288065 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:35.288351 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:35.288499 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:35.296974 (+ 8475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:38721: BlockingRecv error: recv got EOF from 127.4.221.66:38721 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 07 02:22:35 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:35 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:35.488336 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:35.481245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:59723)
0807 02:22:35.481752 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:35.481775 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:35.482031 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:35.482483 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:35.482492 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:35.482852 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:35.483037 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:35.483055 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:35.483927 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:35.483930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:35.484518 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:35.484526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:35.484992 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:35.484995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:35.485065 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:35.485766 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:35.485788 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:35.488180 (+ 2392us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":179,"threads_started":1}
W20260807 02:22:35.488579 14450 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:35.481520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:59723 (local address 127.4.221.126:32833)
0807 02:22:35.481752 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:35.481759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:35.481936 (+ 177us) server_negotiation.cc:407] Connection header received
0807 02:22:35.482186 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:35.482189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:35.482292 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:35.482394 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:35.483175 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:35.483787 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:35.484649 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:35.484883 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:35.484977 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:35.488382 (+ 3405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:59723: BlockingRecv error: recv got EOF from 127.4.221.67:59723 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Aug 07 02:22:36 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:36 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:36.080772 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:36.074049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:53651)
0807 02:22:36.074421 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:36.074451 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:36.074598 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:36.074976 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:36.074978 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:36.075245 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:36.075390 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:36.075397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:36.076346 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:36.076353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:36.077069 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:36.077081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:36.077549 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:36.077551 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:36.077620 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:36.078356 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:36.078370 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:36.080607 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":150,"threads_started":1}
W20260807 02:22:36.080957 14455 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:36.074451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:53651 (local address 127.4.221.126:32833)
0807 02:22:36.074734 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:36.074738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:36.074750 (+ 12us) server_negotiation.cc:407] Connection header received
0807 02:22:36.074789 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:36.074791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:36.074842 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:36.074930 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:36.075533 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:36.076177 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:36.077184 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:36.077430 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:36.077511 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:36.080795 (+ 3284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:53651: BlockingRecv error: recv got EOF from 127.4.221.65:53651 (error 108)
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":104,"threads_started":1}
Aug 07 02:22:36 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:36 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 02:22:36.306998 14455 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:36.298206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:42013 (local address 127.4.221.126:32833)
0807 02:22:36.298462 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:36.298467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:36.298592 (+ 125us) server_negotiation.cc:407] Connection header received
0807 02:22:36.298765 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:36.298817 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:36.298917 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:36.299004 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:36.299679 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:36.300642 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:36.302008 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:36.302355 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:36.302477 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:36.306877 (+ 4400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:42013: BlockingRecv error: recv got EOF from 127.4.221.66:42013 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260807 02:22:36.307112 14456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:36.298020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:42013)
0807 02:22:36.298485 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:36.298501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:36.298636 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:36.299083 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:36.299087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:36.299360 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:36.299536 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:36.299545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:36.300808 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:36.300811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:36.301857 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:36.301869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:36.302497 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:36.302500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:36.302607 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:36.303418 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:36.303435 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:36.306715 (+ 3280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":124,"threads_started":1}
Aug 07 02:22:36 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:36 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:36.496176 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:36.489646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:52329)
0807 02:22:36.490010 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:36.490029 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:36.490183 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:36.490596 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:36.490600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:36.490876 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:36.491056 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:36.491067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:36.491954 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:36.491957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:36.492568 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:36.492576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:36.493033 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:36.493036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:36.493114 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:36.493736 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:36.493748 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:36.496033 (+ 2285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":113,"threads_started":1}
W20260807 02:22:36.496378 14455 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:36.489726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:52329 (local address 127.4.221.126:32833)
0807 02:22:36.489936 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:36.489941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:36.490119 (+ 178us) server_negotiation.cc:407] Connection header received
0807 02:22:36.490306 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:36.490311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:36.490383 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:36.490493 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:36.491204 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:36.491829 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:36.492691 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:36.492918 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:36.493000 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:36.496195 (+ 3195us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:52329: BlockingRecv error: recv got EOF from 127.4.221.67:52329 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 07 02:22:37 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:37 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:37.090430 14458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:37.082079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:54973)
0807 02:22:37.082485 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:37.082522 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:37.082732 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:37.083112 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:37.083115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:37.083398 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:37.083580 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:37.083588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:37.084795 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:37.084799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:37.085577 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:37.085590 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:37.086410 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:37.086414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:37.086508 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:37.087222 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:37.087240 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:37.090185 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":114,"threads_started":1}
W20260807 02:22:37.090834 14459 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:37.082147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:54973 (local address 127.4.221.126:32833)
0807 02:22:37.082520 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:37.082524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:37.082639 (+ 115us) server_negotiation.cc:407] Connection header received
0807 02:22:37.082860 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:37.082864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:37.082943 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:37.083080 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:37.083743 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:37.084666 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:37.085963 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:37.086273 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:37.086397 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:37.090446 (+ 4049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:54973: BlockingRecv error: recv got EOF from 127.4.221.65:54973 (error 108)
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":84,"threads_started":1}
Aug 07 02:22:37 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:37 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:37.314884 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:37.308185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:50121)
0807 02:22:37.308596 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:37.308613 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:37.308782 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:37.309097 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:37.309101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:37.309381 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:37.309516 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:37.309522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:37.310464 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:37.310471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:37.311078 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:37.311087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:37.311573 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:37.311577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:37.311667 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:37.312297 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:37.312310 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:37.314682 (+ 2372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":124,"threads_started":1}
W20260807 02:22:37.314954 14459 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:37.308239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:50121 (local address 127.4.221.126:32833)
0807 02:22:37.308436 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:37.308442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:37.308707 (+ 265us) server_negotiation.cc:407] Connection header received
0807 02:22:37.308905 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:37.308913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:37.308972 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:37.309065 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:37.309648 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:37.310342 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:37.311205 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:37.311446 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:37.311569 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:37.314831 (+ 3262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:50121: BlockingRecv error: recv got EOF from 127.4.221.66:50121 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 07 02:22:37 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:37 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:37.503780 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:37.497178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:55081)
0807 02:22:37.497592 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:37.497607 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:37.497732 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:37.498125 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:37.498129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:37.498369 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:37.498529 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:37.498536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:37.499405 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:37.499410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:37.500103 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:37.500118 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:37.500644 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:37.500647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:37.500712 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:37.501359 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:37.501371 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:37.503608 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":164,"threads_started":1}
W20260807 02:22:37.503965 14459 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:37.497284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:55081 (local address 127.4.221.126:32833)
0807 02:22:37.497517 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:37.497523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:37.497691 (+ 168us) server_negotiation.cc:407] Connection header received
0807 02:22:37.497885 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:37.497889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:37.497963 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:37.498040 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:37.498661 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:37.499285 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:37.500295 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:37.500540 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:37.500622 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:37.503800 (+ 3178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:55081: BlockingRecv error: recv got EOF from 127.4.221.67:55081 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 07 02:22:38 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:38 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
W20260807 02:22:38.098531 14463 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:38.091702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:40901 (local address 127.4.221.126:32833)
0807 02:22:38.092042 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:38.092047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:38.092180 (+ 133us) server_negotiation.cc:407] Connection header received
0807 02:22:38.092303 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:38.092305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:38.092364 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:38.092461 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:38.093071 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:38.093709 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:38.094612 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:38.094859 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:38.094958 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:38.098311 (+ 3353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:40901: BlockingRecv error: recv got EOF from 127.4.221.65:40901 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":108,"threads_started":1}
I20260807 02:22:38.098846 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:38.091655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:40901)
0807 02:22:38.092062 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:38.092078 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:38.092207 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:38.092537 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:38.092539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:38.092785 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:38.092930 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:38.092937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:38.093845 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:38.093848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:38.094493 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:38.094502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:38.094981 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:38.094983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:38.095054 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:38.095673 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:38.095686 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:38.098698 (+ 3012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":138,"threads_started":1}
Aug 07 02:22:38 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:38 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:38.323208 14464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:38.316167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:47861)
0807 02:22:38.316635 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:38.316657 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:38.316854 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:38.317239 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:38.317242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:38.317531 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:38.317663 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:38.317670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:38.318652 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:38.318659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:38.319331 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:38.319340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:38.319903 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:38.319907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:38.320024 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:38.320729 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:38.320743 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:38.323066 (+ 2323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":114,"threads_started":1}
W20260807 02:22:38.323375 14463 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:38.316328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:47861 (local address 127.4.221.126:32833)
0807 02:22:38.316639 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:38.316646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:38.316767 (+ 121us) server_negotiation.cc:407] Connection header received
0807 02:22:38.317019 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:38.317023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:38.317092 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:38.317204 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:38.317842 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:38.318519 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:38.319472 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:38.319758 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:38.319896 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:38.323242 (+ 3346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:47861: BlockingRecv error: recv got EOF from 127.4.221.66:47861 (error 108)
Metrics: {"server-negotiator.queue_time_us":182}
Aug 07 02:22:38 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:38 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:38.511831 14466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:38.504908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:34197)
0807 02:22:38.505279 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:38.505295 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:38.505408 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:38.505747 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:38.505749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:38.506023 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:38.506147 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:38.506153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:38.507033 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:38.507036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:38.507792 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:38.507802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:38.508311 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:38.508314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:38.508423 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:38.509120 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:38.509139 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:38.511625 (+ 2486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":153,"threads_started":1}
W20260807 02:22:38.511875 14463 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:38.505108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:34197 (local address 127.4.221.126:32833)
0807 02:22:38.505269 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:38.505273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:38.505370 (+ 97us) server_negotiation.cc:407] Connection header received
0807 02:22:38.505549 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:38.505553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:38.505614 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:38.505693 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:38.506289 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:38.506910 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:38.507940 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:38.508194 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:38.508287 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:38.511761 (+ 3474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:34197: BlockingRecv error: recv got EOF from 127.4.221.67:34197 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 07 02:22:39 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:39 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:39.106978 14467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:39.100078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:53265)
0807 02:22:39.100481 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:39.100500 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:39.100658 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:39.101030 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:39.101033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:39.101356 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:39.101519 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:39.101526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:39.102433 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:39.102436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:39.103042 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:39.103053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:39.103643 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:39.103646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:39.103735 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:39.104457 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:39.104471 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:39.106817 (+ 2346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":163,"threads_started":1}
W20260807 02:22:39.107220 14468 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:39.100194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:53265 (local address 127.4.221.126:32833)
0807 02:22:39.100481 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:39.100486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:39.100599 (+ 113us) server_negotiation.cc:407] Connection header received
0807 02:22:39.100770 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:39.100773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:39.100845 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:39.100976 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:39.101643 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:39.102282 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:39.103201 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:39.103509 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:39.103614 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:39.107032 (+ 3418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:53265: BlockingRecv error: recv got EOF from 127.4.221.65:53265 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
Aug 07 02:22:39 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:39 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:39.331454 14469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:39.324380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:38017)
0807 02:22:39.324800 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:39.324826 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:39.324980 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:39.325325 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:39.325328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:39.325653 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:39.325791 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:39.325798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:39.326679 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:39.326683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:39.327342 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:39.327351 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:39.327804 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:39.327808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:39.327884 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:39.328521 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:39.328535 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:39.331223 (+ 2688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":133,"threads_started":1}
W20260807 02:22:39.331562 14468 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:39.324404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:38017 (local address 127.4.221.126:32833)
0807 02:22:39.324582 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:39.324586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:39.324914 (+ 328us) server_negotiation.cc:407] Connection header received
0807 02:22:39.325093 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:39.325099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:39.325173 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:39.325294 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:39.325977 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:39.326568 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:39.327455 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:39.327694 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:39.327776 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:39.331416 (+ 3640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:38017: BlockingRecv error: recv got EOF from 127.4.221.66:38017 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 07 02:22:39 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:39 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:39.519109 14470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:39.512722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:53689)
0807 02:22:39.513086 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:39.513101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:39.513225 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:39.513557 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:39.513559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:39.513792 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:39.513933 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:39.513939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:39.514766 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:39.514769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:39.515348 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:39.515355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:39.515845 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:39.515848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:39.515916 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:39.516564 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:39.516577 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:39.518952 (+ 2375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":170,"threads_started":1}
W20260807 02:22:39.519249 14468 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:39.512800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:53689 (local address 127.4.221.126:32833)
0807 02:22:39.512986 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:39.512991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:39.513195 (+ 204us) server_negotiation.cc:407] Connection header received
0807 02:22:39.513316 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:39.513318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:39.513382 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:39.513492 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:39.514047 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:39.514671 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:39.515479 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:39.515744 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:39.515848 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:39.519115 (+ 3267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:53689: BlockingRecv error: recv got EOF from 127.4.221.67:53689 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 07 02:22:40 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:40 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:40.114822 14471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:40.108053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:59201)
0807 02:22:40.108414 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:40.108434 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:40.108585 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:40.108907 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:40.108910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:40.109175 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:40.109303 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:40.109310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:40.110228 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:40.110233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:40.110840 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:40.110849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:40.111368 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:40.111372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:40.111483 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:40.112133 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:40.112147 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:40.114604 (+ 2457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":106,"threads_started":1}
W20260807 02:22:40.114934 14472 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:40.108071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:59201 (local address 127.4.221.126:32833)
0807 02:22:40.108447 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:40.108451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:40.108511 (+ 60us) server_negotiation.cc:407] Connection header received
0807 02:22:40.108704 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:40.108708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:40.108771 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:40.108869 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:40.109430 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:40.110116 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:40.110972 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:40.111244 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:40.111361 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:40.114778 (+ 3417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:59201: BlockingRecv error: recv got EOF from 127.4.221.65:59201 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":121,"threads_started":1}
Aug 07 02:22:40 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:40 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:40.344386 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:40.332722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:44127)
0807 02:22:40.333104 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:40.333120 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:40.333254 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:40.333589 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:40.333593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:40.333959 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:40.334097 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:40.334105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:40.335016 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:40.335022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:40.335703 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:40.335711 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:40.336256 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:40.336260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:40.336369 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:40.341794 (+ 5425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:40.341812 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:40.344220 (+ 2408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":155,"threads_started":1}
W20260807 02:22:40.344666 14472 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:40.332802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:44127 (local address 127.4.221.126:32833)
0807 02:22:40.332973 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:40.332977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:40.333207 (+ 230us) server_negotiation.cc:407] Connection header received
0807 02:22:40.333381 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:40.333385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:40.333453 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:40.333548 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:40.334233 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:40.334895 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:40.335835 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:40.336124 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:40.336240 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:40.344507 (+ 8267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:44127: BlockingRecv error: recv got EOF from 127.4.221.66:44127 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 07 02:22:40 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:40 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:40.527254 14474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:40.520196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:38175)
0807 02:22:40.520584 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:40.520600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:40.520736 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:40.521027 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:40.521030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:40.521281 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:40.521452 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:40.521460 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:40.522367 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:40.522371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:40.523006 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:40.523020 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:40.523622 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:40.523629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:40.523739 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:40.524538 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:40.524557 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:40.527043 (+ 2486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":101,"threads_started":1}
W20260807 02:22:40.527370 14472 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:40.520356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:38175 (local address 127.4.221.126:32833)
0807 02:22:40.520591 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:40.520595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:40.520666 (+ 71us) server_negotiation.cc:407] Connection header received
0807 02:22:40.520835 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:40.520837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:40.520894 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:40.520972 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:40.521579 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:40.522241 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:40.523151 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:40.523451 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:40.523576 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:40.527221 (+ 3645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:38175: BlockingRecv error: recv got EOF from 127.4.221.67:38175 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Aug 07 02:22:41 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:41 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:41.122784 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:41.115953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:34385)
0807 02:22:41.116570 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:41.116585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:41.116711 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:41.116981 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:41.116983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:41.117219 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:41.117347 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:41.117353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:41.118282 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:41.118285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:41.118937 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:41.118947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:41.119543 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:41.119546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:41.119619 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:41.120356 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:41.120370 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:41.122587 (+ 2217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":123,"threads_started":1}
W20260807 02:22:41.123025 14476 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:41.116071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:34385 (local address 127.4.221.126:32833)
0807 02:22:41.116418 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:41.116423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:41.116665 (+ 242us) server_negotiation.cc:407] Connection header received
0807 02:22:41.116802 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:41.116805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:41.116861 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:41.116959 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:41.117479 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:41.118155 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:41.119094 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:41.119363 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:41.119459 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:41.122818 (+ 3359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:34385: BlockingRecv error: recv got EOF from 127.4.221.65:34385 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":117,"threads_started":1}
Aug 07 02:22:41 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:41 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:41.352789 14477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:41.345594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:48359)
0807 02:22:41.346029 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:41.346049 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:41.346222 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:41.346565 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:41.346570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:41.346858 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:41.346993 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:41.347000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:41.347954 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:41.347958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:41.348614 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:41.348625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:41.349172 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:41.349176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:41.349266 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:41.350177 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:41.350192 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:41.352630 (+ 2438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":163,"threads_started":1}
W20260807 02:22:41.353073 14476 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:41.345777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:48359 (local address 127.4.221.126:32833)
0807 02:22:41.346026 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:41.346032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:41.346139 (+ 107us) server_negotiation.cc:407] Connection header received
0807 02:22:41.346353 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:41.346357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:41.346424 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:41.346533 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:41.347126 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:41.347812 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:41.348745 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:41.349028 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:41.349123 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:41.352789 (+ 3666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:48359: BlockingRecv error: recv got EOF from 127.4.221.66:48359 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
Aug 07 02:22:41 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:41 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:41.535465 14478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:41.528320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:33427)
0807 02:22:41.528741 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:41.528757 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:41.528942 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:41.529301 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:41.529305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:41.529596 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:41.529739 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:41.529747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:41.530802 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:41.530809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:41.531450 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:41.531460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:41.531963 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:41.531967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:41.532064 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:41.532835 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:41.532850 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:41.535211 (+ 2361us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":169,"threads_started":1}
W20260807 02:22:41.535521 14476 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:41.528405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:33427 (local address 127.4.221.126:32833)
0807 02:22:41.528624 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:41.528631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:41.528858 (+ 227us) server_negotiation.cc:407] Connection header received
0807 02:22:41.529089 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:41.529093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:41.529160 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:41.529273 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:41.529954 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:41.530675 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:41.531587 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:41.531827 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:41.531958 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:41.535384 (+ 3426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:33427: BlockingRecv error: recv got EOF from 127.4.221.67:33427 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 07 02:22:42 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:42 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:42.131312 14479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:42.124163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:48489)
0807 02:22:42.124589 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:42.124612 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:42.124792 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:42.125162 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:42.125165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:42.125464 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:42.125593 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:42.125601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:42.126662 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:42.126668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:42.127390 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:42.127400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:42.127957 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:42.127961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:42.128063 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:42.128814 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:42.128829 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:42.131146 (+ 2317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":140,"threads_started":1}
W20260807 02:22:42.131582 14480 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:42.124238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:48489 (local address 127.4.221.126:32833)
0807 02:22:42.124735 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:42.124739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:42.124753 (+ 14us) server_negotiation.cc:407] Connection header received
0807 02:22:42.124935 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:42.124940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:42.125007 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:42.125135 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:42.125772 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:42.126526 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:42.127530 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:42.127816 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:42.127939 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:42.131379 (+ 3440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:48489: BlockingRecv error: recv got EOF from 127.4.221.65:48489 (error 108)
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":148,"threads_started":1}
Aug 07 02:22:42 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:42 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:42.361265 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:42.353975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:55391)
0807 02:22:42.354377 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:42.354397 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:42.354573 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:42.354991 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:42.354995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:42.355351 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:42.355515 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:42.355523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:42.356702 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:42.356706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:42.357496 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:42.357506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:42.358038 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:42.358041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:42.358121 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:42.358780 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:42.358793 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:42.361107 (+ 2314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":138,"threads_started":1}
W20260807 02:22:42.361526 14480 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:42.354081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:55391 (local address 127.4.221.126:32833)
0807 02:22:42.354291 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:42.354298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:42.354526 (+ 228us) server_negotiation.cc:407] Connection header received
0807 02:22:42.354763 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:42.354767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:42.354844 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:42.354950 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:42.355659 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:42.356582 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:42.357638 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:42.357931 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:42.358031 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:42.361334 (+ 3303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:55391: BlockingRecv error: recv got EOF from 127.4.221.66:55391 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 07 02:22:42 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:42 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.67@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:42.543654 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:42.536596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.67:43067)
0807 02:22:42.536992 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:42.537006 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:42.537130 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:42.537449 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:42.537451 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:42.537689 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:42.537874 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:42.537880 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:42.538876 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:42.538879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:42.539514 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:42.539523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:42.540106 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:42.540110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:42.540278 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:42.540959 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:42.540972 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:42.543417 (+ 2445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":180,"threads_started":1}
W20260807 02:22:42.543732 14480 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:42.536646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:43067 (local address 127.4.221.126:32833)
0807 02:22:42.536856 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:42.536882 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:42.537093 (+ 211us) server_negotiation.cc:407] Connection header received
0807 02:22:42.537227 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:42.537230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:42.537299 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:42.537389 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:42.538022 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:42.538741 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:42.539653 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:42.539927 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:42.540026 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:42.543587 (+ 3561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.67:43067: BlockingRecv error: recv got EOF from 127.4.221.67:43067 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 07 02:22:43 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:43 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.65@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:43.139211 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:43.132683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.65:54877)
0807 02:22:43.133104 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:43.133123 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:43.133269 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:43.133642 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:43.133645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:43.133947 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:43.134075 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:43.134082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:43.134969 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:43.134972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:43.135626 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:43.135644 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:43.136099 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:43.136101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:43.136179 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:43.136844 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:43.136857 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:43.139061 (+ 2204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":109,"threads_started":1}
W20260807 02:22:43.139441 14484 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:43.132743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:54877 (local address 127.4.221.126:32833)
0807 02:22:43.133081 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:43.133086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:43.133221 (+ 135us) server_negotiation.cc:407] Connection header received
0807 02:22:43.133380 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:43.133382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:43.133470 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:43.133582 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:43.134190 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:43.134834 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:43.135753 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:43.135989 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:43.136066 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:43.139233 (+ 3167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.65:54877: BlockingRecv error: recv got EOF from 127.4.221.65:54877 (error 108)
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":115,"threads_started":1}
Aug 07 02:22:43 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
Aug 07 02:22:43 dist-test-slave-69rg krb5kdc[13394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.221.66@KRBTEST.COM for oryx/127.4.221.126@KRBTEST.COM, Server not found in Kerberos database
I20260807 02:22:43.369483 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:43.362484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:32833 (local address 127.4.221.66:44065)
0807 02:22:43.362848 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:43.362863 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:43.362978 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:43.363230 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:43.363232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:43.363492 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:43.363618 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:43.363623 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:43.364616 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:43.364620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:43.365337 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:43.365349 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:43.366001 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:43.366005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:43.366091 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:43.366762 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:43.366776 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:43.369230 (+ 2454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.221.126:32833: Server oryx/127.4.221.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":159,"threads_started":1}
W20260807 02:22:43.369568 14484 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:43.362527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:44065 (local address 127.4.221.126:32833)
0807 02:22:43.362727 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:43.362731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:43.362964 (+ 233us) server_negotiation.cc:407] Connection header received
0807 02:22:43.363062 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:43.363065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:43.363122 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:43.363202 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:43.363753 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:43.364356 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:43.365523 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:43.365769 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:43.365958 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:43.369416 (+ 3458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.221.66:44065: BlockingRecv error: recv got EOF from 127.4.221.66:44065 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 02:22:43.418792 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 13995
I20260807 02:22:43.427278 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 14151
I20260807 02:22:43.434623 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 14285
I20260807 02:22:43.440856 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 13920
2026-08-07T02:22:43Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18234 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-07T02:22:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:43Z Disabled control of system clock
I20260807 02:22:43.478045 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:37385
--webserver_interface=127.4.221.126
--webserver_port=0
--builtin_ntp_servers=127.4.221.84:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:37385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260807 02:22:43.597970 14489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:43.598301 14489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:43.598340 14489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:43.602571 14489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:43.602643 14489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:43.602660 14489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:43.602679 14489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:43.602694 14489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:22:43.607784 14489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:37385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:37385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:43.608999 14489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:43.610468 14489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 02:22:43.619707 14489 server_base.cc:1034] running on GCE node
W20260807 02:22:43.619841 14495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:43.619947 14497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:43.619848 14494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:43.620568 14489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:43.621752 14489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:43.622963 14489 hybrid_clock.cc:648] HybridClock initialized: now 1786069363622935 us; error 44 us; skew 500 ppm
I20260807 02:22:43.625056 14489 webserver.cc:460] Webserver started at http://127.4.221.126:39017/ using document root <none> and password file <none>
I20260807 02:22:43.625677 14489 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:43.625771 14489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:43.626025 14489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:43.628165 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "11c1f050a4b345e8afc19ae3667b5109"
format_stamp: "Formatted at 2026-08-07 02:22:43 on dist-test-slave-69rg"
server_key: "a15f5ce44cb0575c39eada0580be0a0b"
server_key_iv: "1cc911471209941b942ba31a0bd33643"
server_key_version: "encryptionkey@0"
I20260807 02:22:43.628747 14489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "11c1f050a4b345e8afc19ae3667b5109"
format_stamp: "Formatted at 2026-08-07 02:22:43 on dist-test-slave-69rg"
server_key: "a15f5ce44cb0575c39eada0580be0a0b"
server_key_iv: "1cc911471209941b942ba31a0bd33643"
server_key_version: "encryptionkey@0"
I20260807 02:22:43.632566 14489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260807 02:22:43.635053 14503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:43.636354 14489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 02:22:43.636499 14489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "11c1f050a4b345e8afc19ae3667b5109"
format_stamp: "Formatted at 2026-08-07 02:22:43 on dist-test-slave-69rg"
server_key: "a15f5ce44cb0575c39eada0580be0a0b"
server_key_iv: "1cc911471209941b942ba31a0bd33643"
server_key_version: "encryptionkey@0"
I20260807 02:22:43.636615 14489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:43.659114 14489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:43.659966 14489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:43.660198 14489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:43.667706 14489 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:37385
I20260807 02:22:43.667732 14556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:37385 every 8 connection(s)
I20260807 02:22:43.668752 14489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:43.672374 14557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:43.675683 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 14489
I20260807 02:22:43.675823 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:22:43.676148 4981 external_mini_cluster.cc:1428] Setting key 8b7576ce669a7d7613c0f02faa942021
I20260807 02:22:43.679160 14557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109: Bootstrap starting.
I20260807 02:22:43.681913 14557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:43.682850 14557 log.cc:824] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:43.685420 14557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109: No bootstrap required, opened a new log
I20260807 02:22:43.686476 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:43.677718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.4.221.126:37385)
0807 02:22:43.678441 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:43.678459 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:43.678497 (+ 38us) server_negotiation.cc:407] Connection header received
0807 02:22:43.679174 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:43.679191 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:43.679639 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:43.679928 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:43.680296 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:43.681271 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:43.682560 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:43.683128 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:43.683311 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:43.684229 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:43.684236 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:43.684253 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:43.684765 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:43.684838 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:43.684864 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:43.684968 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:43.685120 (+ 152us) server_negotiation.cc:299] Negotiation successful
0807 02:22:43.685411 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":133,"threads_started":1}
I20260807 02:22:43.690024 14557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c1f050a4b345e8afc19ae3667b5109" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37385 } }
I20260807 02:22:43.690264 14557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:43.690299 14557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c1f050a4b345e8afc19ae3667b5109, State: Initialized, Role: FOLLOWER
I20260807 02:22:43.690677 14557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c1f050a4b345e8afc19ae3667b5109" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37385 } }
I20260807 02:22:43.690799 14557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:43.690846 14557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:43.690919 14557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:43.691882 14557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c1f050a4b345e8afc19ae3667b5109" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37385 } }
I20260807 02:22:43.692184 14557 leader_election.cc:302] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11c1f050a4b345e8afc19ae3667b5109; no voters:
I20260807 02:22:43.692476 14557 leader_election.cc:288] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:22:43.692601 14562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:43.692862 14562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [term 1 LEADER]: Becoming Leader. State: Replica: 11c1f050a4b345e8afc19ae3667b5109, State: Running, Role: LEADER
I20260807 02:22:43.693257 14562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c1f050a4b345e8afc19ae3667b5109" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37385 } }
I20260807 02:22:43.693889 14557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:43.695119 14564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11c1f050a4b345e8afc19ae3667b5109. Latest consensus state: current_term: 1 leader_uuid: "11c1f050a4b345e8afc19ae3667b5109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c1f050a4b345e8afc19ae3667b5109" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37385 } } }
I20260807 02:22:43.695119 14563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11c1f050a4b345e8afc19ae3667b5109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c1f050a4b345e8afc19ae3667b5109" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37385 } } }
I20260807 02:22:43.695405 14563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:43.695396 14564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:43.695847 14571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:43.699208 14571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:43.705245 14571 catalog_manager.cc:1347] Generated new cluster ID: 146a1bff8e5c45c68158373ca5acd20e
I20260807 02:22:43.705348 14571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:43.722894 14571 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:22:43.723968 14571 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:43.734429 14571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11c1f050a4b345e8afc19ae3667b5109: Generated new TSK 0
I20260807 02:22:43.735383 14571 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:22:43.748147 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:37385
--builtin_ntp_servers=127.4.221.84:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260807 02:22:43.868453 14581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:43.868759 14581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:43.868798 14581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:43.873071 14581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:43.873165 14581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:43.873224 14581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:22:43.878382 14581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:37385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:43.879781 14581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:43.881423 14581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:43.891521 14586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:43.891554 14587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:43.891521 14589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:43.891618 14581 server_base.cc:1034] running on GCE node
I20260807 02:22:43.892300 14581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:43.893260 14581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:43.894423 14581 hybrid_clock.cc:648] HybridClock initialized: now 1786069363894397 us; error 42 us; skew 500 ppm
I20260807 02:22:43.896828 14581 webserver.cc:460] Webserver started at http://127.4.221.65:35591/ using document root <none> and password file <none>
I20260807 02:22:43.897390 14581 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:43.897454 14581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:43.897635 14581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:43.899852 14581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "10b3d5cbac1448ac8741d79762040edf"
format_stamp: "Formatted at 2026-08-07 02:22:43 on dist-test-slave-69rg"
server_key: "b01d65ece8d03fb437612d963c8a8960"
server_key_iv: "ea343111aa4ede7a88c4f70a5d562c32"
server_key_version: "encryptionkey@0"
I20260807 02:22:43.900385 14581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "10b3d5cbac1448ac8741d79762040edf"
format_stamp: "Formatted at 2026-08-07 02:22:43 on dist-test-slave-69rg"
server_key: "b01d65ece8d03fb437612d963c8a8960"
server_key_iv: "ea343111aa4ede7a88c4f70a5d562c32"
server_key_version: "encryptionkey@0"
I20260807 02:22:43.904242 14581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:22:43.907064 14595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:43.908414 14581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:22:43.908573 14581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "10b3d5cbac1448ac8741d79762040edf"
format_stamp: "Formatted at 2026-08-07 02:22:43 on dist-test-slave-69rg"
server_key: "b01d65ece8d03fb437612d963c8a8960"
server_key_iv: "ea343111aa4ede7a88c4f70a5d562c32"
server_key_version: "encryptionkey@0"
I20260807 02:22:43.908705 14581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:43.923802 14581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:43.924608 14581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:43.924839 14581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:43.925490 14581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:43.926630 14581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:43.926705 14581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:43.926774 14581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:43.926821 14581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:43.935966 14581 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:42311
I20260807 02:22:43.936036 14710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:42311 every 8 connection(s)
I20260807 02:22:43.937049 14581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:22:43.946527 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 14581
I20260807 02:22:43.946663 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:22:43.946985 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:43.939139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:40277 (local address 127.4.221.126:37385)
0807 02:22:43.939288 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:43.939292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:43.940415 (+ 1123us) server_negotiation.cc:407] Connection header received
0807 02:22:43.941945 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:43.941948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:43.941998 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:43.942075 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:43.942975 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:43.943549 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:43.944730 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:43.944952 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:43.945038 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:43.946374 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:43.946377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:43.946380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:43.946401 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:43.946421 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:43.946428 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:43.946506 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:43.946737 (+ 231us) server_negotiation.cc:299] Negotiation successful
0807 02:22:43.946814 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 02:22:43.947033 4981 external_mini_cluster.cc:1428] Setting key 9a374fc6c2fa159e1d4b07bc16a0a34a
I20260807 02:22:43.947986 14713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:43.939543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:37385 (local address 127.4.221.65:40277)
0807 02:22:43.940273 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:43.940316 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:43.941672 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:43.942238 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:43.942245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:43.942330 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:43.942843 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:43.942856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:43.943662 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:43.943665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:43.944604 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:43.944615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:43.945122 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:43.945127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:43.945327 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:43.946083 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:22:43.946085 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:43.946129 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:22:43.946204 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:43.946508 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:43.946522 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:43.946528 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 02:22:43.946674 (+ 146us) client_negotiation.cc:240] Negotiation successful
0807 02:22:43.946957 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":125,"threads_started":1}
I20260807 02:22:43.949132 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:37385
--builtin_ntp_servers=127.4.221.84:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 02:22:43.949249 14711 heartbeater.cc:347] Connected to a master server at 127.4.221.126:37385
I20260807 02:22:43.949497 14711 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:43.950178 14711 heartbeater.cc:511] Master 127.4.221.126:37385 requested a full tablet report, sending...
I20260807 02:22:43.951707 14520 ts_manager.cc:194] Registered new tserver with Master: 10b3d5cbac1448ac8741d79762040edf (127.4.221.65:42311)
I20260807 02:22:43.952850 14520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.221.65:40277
W20260807 02:22:44.068917 14714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:44.069350 14714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:44.069424 14714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:44.074254 14714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:44.074343 14714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:44.074431 14714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:22:44.079756 14714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:37385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:44.081038 14714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:44.082641 14714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:44.092799 14722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:44.092808 14720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:44.092820 14719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:44.093021 14714 server_base.cc:1034] running on GCE node
I20260807 02:22:44.093571 14714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:44.094386 14714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:44.095547 14714 hybrid_clock.cc:648] HybridClock initialized: now 1786069364095522 us; error 43 us; skew 500 ppm
I20260807 02:22:44.097560 14714 webserver.cc:460] Webserver started at http://127.4.221.66:40787/ using document root <none> and password file <none>
I20260807 02:22:44.098240 14714 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:44.098311 14714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:44.098554 14714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:44.100796 14714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "073c171e56954cc1bf0f811e9c50dfb5"
format_stamp: "Formatted at 2026-08-07 02:22:44 on dist-test-slave-69rg"
server_key: "14285df051f7bac110aafb1f93b2efeb"
server_key_iv: "4ec91fb5fe5132427274b8e102fa38f3"
server_key_version: "encryptionkey@0"
I20260807 02:22:44.101416 14714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "073c171e56954cc1bf0f811e9c50dfb5"
format_stamp: "Formatted at 2026-08-07 02:22:44 on dist-test-slave-69rg"
server_key: "14285df051f7bac110aafb1f93b2efeb"
server_key_iv: "4ec91fb5fe5132427274b8e102fa38f3"
server_key_version: "encryptionkey@0"
I20260807 02:22:44.105471 14714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260807 02:22:44.108302 14728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:44.109599 14714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:22:44.109731 14714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "073c171e56954cc1bf0f811e9c50dfb5"
format_stamp: "Formatted at 2026-08-07 02:22:44 on dist-test-slave-69rg"
server_key: "14285df051f7bac110aafb1f93b2efeb"
server_key_iv: "4ec91fb5fe5132427274b8e102fa38f3"
server_key_version: "encryptionkey@0"
I20260807 02:22:44.109915 14714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:44.132325 14714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:44.133152 14714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:44.133374 14714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:44.134136 14714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:44.135182 14714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:44.135255 14714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:44.135327 14714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:44.135371 14714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:44.145010 14714 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:44783
I20260807 02:22:44.145031 14842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:44783 every 8 connection(s)
I20260807 02:22:44.146056 14714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:22:44.147456 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 14714
I20260807 02:22:44.147588 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:22:44.147907 4981 external_mini_cluster.cc:1428] Setting key 3e0277da7bdd90eb3a80d135b998c5c1
I20260807 02:22:44.149981 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:37385
--builtin_ntp_servers=127.4.221.84:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260807 02:22:44.156673 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:44.148046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:33767 (local address 127.4.221.126:37385)
0807 02:22:44.148213 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:44.148218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:44.149471 (+ 1253us) server_negotiation.cc:407] Connection header received
0807 02:22:44.151033 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:44.151039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:44.151103 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:44.151200 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:44.152142 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:44.153021 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:44.154302 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:44.154707 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:44.154845 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:44.156026 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:44.156033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:44.156039 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:44.156060 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:44.156088 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:44.156103 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:44.156188 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:44.156357 (+ 169us) server_negotiation.cc:299] Negotiation successful
0807 02:22:44.156445 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 02:22:44.157583 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:44.148569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:37385 (local address 127.4.221.66:33767)
0807 02:22:44.149313 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:44.149367 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:44.150827 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:44.151368 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:44.151376 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:44.151461 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:44.151993 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:44.152006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:44.153157 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:44.153159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:44.154174 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:44.154185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:44.154877 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:44.154880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:44.155038 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:44.155733 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:22:44.155735 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:44.155780 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:22:44.155852 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:44.156212 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:44.156220 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:44.156225 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 02:22:44.156327 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 02:22:44.156624 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":140,"threads_started":1}
I20260807 02:22:44.158938 14843 heartbeater.cc:347] Connected to a master server at 127.4.221.126:37385
I20260807 02:22:44.159190 14843 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:44.159771 14843 heartbeater.cc:511] Master 127.4.221.126:37385 requested a full tablet report, sending...
I20260807 02:22:44.160919 14520 ts_manager.cc:194] Registered new tserver with Master: 073c171e56954cc1bf0f811e9c50dfb5 (127.4.221.66:44783)
I20260807 02:22:44.161579 14520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.221.66:33767
W20260807 02:22:44.272650 14846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:44.272979 14846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:44.273048 14846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:44.277740 14846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:44.277850 14846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:44.277951 14846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:22:44.282888 14846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:37385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:44.284078 14846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:44.285583 14846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:44.295981 14851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:44.295979 14854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:44.295975 14852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:44.296798 14846 server_base.cc:1034] running on GCE node
I20260807 02:22:44.297299 14846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:44.298105 14846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:44.299304 14846 hybrid_clock.cc:648] HybridClock initialized: now 1786069364299291 us; error 38 us; skew 500 ppm
I20260807 02:22:44.301257 14846 webserver.cc:460] Webserver started at http://127.4.221.67:46369/ using document root <none> and password file <none>
I20260807 02:22:44.301939 14846 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:44.302037 14846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:44.302277 14846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:44.304476 14846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "22f059092778453ca0bba9a9b9a2ca1f"
format_stamp: "Formatted at 2026-08-07 02:22:44 on dist-test-slave-69rg"
server_key: "ef1c2a30f6939ab40aceabe52117f6da"
server_key_iv: "a0adb236fbe5333cf080752f1948c524"
server_key_version: "encryptionkey@0"
I20260807 02:22:44.305110 14846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "22f059092778453ca0bba9a9b9a2ca1f"
format_stamp: "Formatted at 2026-08-07 02:22:44 on dist-test-slave-69rg"
server_key: "ef1c2a30f6939ab40aceabe52117f6da"
server_key_iv: "a0adb236fbe5333cf080752f1948c524"
server_key_version: "encryptionkey@0"
I20260807 02:22:44.309187 14846 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 02:22:44.312117 14860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:44.313783 14846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:22:44.313972 14846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "22f059092778453ca0bba9a9b9a2ca1f"
format_stamp: "Formatted at 2026-08-07 02:22:44 on dist-test-slave-69rg"
server_key: "ef1c2a30f6939ab40aceabe52117f6da"
server_key_iv: "a0adb236fbe5333cf080752f1948c524"
server_key_version: "encryptionkey@0"
I20260807 02:22:44.314116 14846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:44.332451 14846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:44.333361 14846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:44.333602 14846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:44.334343 14846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:44.335469 14846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:44.335544 14846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:44.335623 14846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:44.335673 14846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:44.346184 14846 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:46167
I20260807 02:22:44.346210 14974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:46167 every 8 connection(s)
I20260807 02:22:44.347246 14846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:22:44.348244 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 14846
I20260807 02:22:44.348363 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:22:44.348688 4981 external_mini_cluster.cc:1428] Setting key c536001adcb9b09e20e481cf0b3ddcf0
I20260807 02:22:44.358280 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:44.349510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:53809 (local address 127.4.221.126:37385)
0807 02:22:44.349686 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:44.349691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:44.351219 (+ 1528us) server_negotiation.cc:407] Connection header received
0807 02:22:44.352815 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:44.352817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:44.352879 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:44.352955 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:44.354092 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:44.354685 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:44.356000 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:44.356270 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:44.356370 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:44.357694 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:44.357697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:44.357699 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:44.357716 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:44.357737 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:44.357745 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:44.357811 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:44.358036 (+ 225us) server_negotiation.cc:299] Negotiation successful
0807 02:22:44.358107 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 02:22:44.359292 14977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:44.350095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:37385 (local address 127.4.221.67:53809)
0807 02:22:44.351020 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:44.351085 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:44.352586 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:44.353178 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:44.353186 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:44.353280 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:44.353925 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:44.353947 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:44.354840 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:44.354843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:44.355846 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:44.355863 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:44.356462 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:44.356465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:44.356633 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:44.357398 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:22:44.357399 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:44.357448 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:22:44.357527 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:44.357888 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:44.357899 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:44.357908 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 02:22:44.358023 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 02:22:44.358351 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":165,"threads_started":1}
I20260807 02:22:44.360623 14975 heartbeater.cc:347] Connected to a master server at 127.4.221.126:37385
I20260807 02:22:44.360904 14975 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:44.361562 14975 heartbeater.cc:511] Master 127.4.221.126:37385 requested a full tablet report, sending...
I20260807 02:22:44.362830 14520 ts_manager.cc:194] Registered new tserver with Master: 22f059092778453ca0bba9a9b9a2ca1f (127.4.221.67:46167)
I20260807 02:22:44.363283 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:22:44.363569 14520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.221.67:53809
W20260807 02:22:44.366768 14560 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:44.365603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.4.221.126:37385)
0807 02:22:44.365740 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:44.365744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:44.365993 (+ 249us) server_negotiation.cc:407] Connection header received
0807 02:22:44.366092 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:44.366094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:44.366138 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:44.366210 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:44.366592 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43192: BlockingRecv error: recv got EOF from 127.0.0.1:43192 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 02:22:44.366859 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 14581
I20260807 02:22:44.375839 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 14714
I20260807 02:22:44.383582 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 14846
I20260807 02:22:44.390363 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 14489
2026-08-07T02:22:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (947 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-07T02:22:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:44Z Disabled control of system clock
I20260807 02:22:44.423815 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:40139
--webserver_interface=127.4.221.126
--webserver_port=0
--builtin_ntp_servers=127.4.221.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:40139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 02:22:44.537657 14986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:44.537997 14986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:44.538036 14986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:44.542187 14986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:44.542259 14986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:44.542275 14986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:44.542291 14986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:44.542307 14986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:22:44.547183 14986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:40139
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:40139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:44.548342 14986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:44.549861 14986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:44.558615 14994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:44.558580 14991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:44.558537 14992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:44.559047 14986 server_base.cc:1034] running on GCE node
I20260807 02:22:44.559614 14986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:44.560765 14986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:44.562018 14986 hybrid_clock.cc:648] HybridClock initialized: now 1786069364562005 us; error 26 us; skew 500 ppm
I20260807 02:22:44.564038 14986 webserver.cc:460] Webserver started at http://127.4.221.126:38725/ using document root <none> and password file <none>
I20260807 02:22:44.564635 14986 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:44.564736 14986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:44.564983 14986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:44.567166 14986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "7077752074bd43019831e71fd5f2de56"
format_stamp: "Formatted at 2026-08-07 02:22:44 on dist-test-slave-69rg"
server_key: "9cee415a8c87e6df5467220b3876533a"
server_key_iv: "5adf5969260c730732cfa25bd3e48787"
server_key_version: "encryptionkey@0"
I20260807 02:22:44.567782 14986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "7077752074bd43019831e71fd5f2de56"
format_stamp: "Formatted at 2026-08-07 02:22:44 on dist-test-slave-69rg"
server_key: "9cee415a8c87e6df5467220b3876533a"
server_key_iv: "5adf5969260c730732cfa25bd3e48787"
server_key_version: "encryptionkey@0"
I20260807 02:22:44.571913 14986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:22:44.574633 15001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:44.576020 14986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:22:44.576182 14986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "7077752074bd43019831e71fd5f2de56"
format_stamp: "Formatted at 2026-08-07 02:22:44 on dist-test-slave-69rg"
server_key: "9cee415a8c87e6df5467220b3876533a"
server_key_iv: "5adf5969260c730732cfa25bd3e48787"
server_key_version: "encryptionkey@0"
I20260807 02:22:44.576327 14986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:44.581182 14986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:44.581866 14986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:44.582050 14986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:44.588734 14986 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:40139
I20260807 02:22:44.588748 15054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:40139 every 8 connection(s)
I20260807 02:22:44.589716 14986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:44.590467 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 14986
I20260807 02:22:44.590600 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:22:44.590934 4981 external_mini_cluster.cc:1428] Setting key b6c46b70a6adccf57e4d0821125c7910
I20260807 02:22:44.593448 15055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:44.597599 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:44.592459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39652 (local address 127.4.221.126:40139)
0807 02:22:44.593747 (+ 1288us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:44.593764 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:44.593807 (+ 43us) server_negotiation.cc:407] Connection header received
0807 02:22:44.594599 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:44.594617 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:44.594978 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:44.595341 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:44.596199 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:44.596204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:44.596215 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:44.596226 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:44.596282 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:44.596299 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:44.596450 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:44.596519 (+ 69us) server_negotiation.cc:299] Negotiation successful
0807 02:22:44.596879 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886,"thread_start_us":717,"threads_started":1}
I20260807 02:22:44.599984 15055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56: Bootstrap starting.
I20260807 02:22:44.602360 15055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:44.603117 15055 log.cc:824] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:44.605067 15055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56: No bootstrap required, opened a new log
I20260807 02:22:44.608842 15055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7077752074bd43019831e71fd5f2de56" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40139 } }
I20260807 02:22:44.609083 15055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:44.609143 15055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7077752074bd43019831e71fd5f2de56, State: Initialized, Role: FOLLOWER
I20260807 02:22:44.609577 15055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7077752074bd43019831e71fd5f2de56" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40139 } }
I20260807 02:22:44.609711 15055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:44.609797 15055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:44.609933 15055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:44.611040 15055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7077752074bd43019831e71fd5f2de56" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40139 } }
I20260807 02:22:44.611421 15055 leader_election.cc:302] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7077752074bd43019831e71fd5f2de56; no voters:
I20260807 02:22:44.611827 15055 leader_election.cc:288] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:22:44.611997 15060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:44.612303 15060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [term 1 LEADER]: Becoming Leader. State: Replica: 7077752074bd43019831e71fd5f2de56, State: Running, Role: LEADER
I20260807 02:22:44.612684 15060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7077752074bd43019831e71fd5f2de56" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40139 } }
I20260807 02:22:44.613219 15055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:44.614605 15062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7077752074bd43019831e71fd5f2de56. Latest consensus state: current_term: 1 leader_uuid: "7077752074bd43019831e71fd5f2de56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7077752074bd43019831e71fd5f2de56" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40139 } } }
I20260807 02:22:44.614646 15061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7077752074bd43019831e71fd5f2de56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7077752074bd43019831e71fd5f2de56" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40139 } } }
I20260807 02:22:44.614743 15062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:44.614753 15061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:44.615196 15069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:44.618510 15069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:44.624446 15069 catalog_manager.cc:1347] Generated new cluster ID: d597772b1ee74e21a9d77294eb3e63aa
I20260807 02:22:44.624528 15069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:44.651657 15069 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:22:44.652029 15069 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:44.666477 15069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7077752074bd43019831e71fd5f2de56: Generated new TSK 0
I20260807 02:22:44.667299 15069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:22:44.671468 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40139
--builtin_ntp_servers=127.4.221.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260807 02:22:44.787545 15080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:44.787897 15080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:44.787974 15080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:44.792269 15080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:44.792344 15080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:44.792405 15080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:22:44.797376 15080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:44.798635 15080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:44.800050 15080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:44.809715 15088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:44.809751 15085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:44.809895 15080 server_base.cc:1034] running on GCE node
W20260807 02:22:44.809765 15086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:44.810470 15080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:44.811211 15080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:44.812382 15080 hybrid_clock.cc:648] HybridClock initialized: now 1786069364812367 us; error 27 us; skew 500 ppm
I20260807 02:22:44.814617 15080 webserver.cc:460] Webserver started at http://127.4.221.65:34593/ using document root <none> and password file <none>
I20260807 02:22:44.815348 15080 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:44.815456 15080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:44.815739 15080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:44.818068 15080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "da55cfa88de049c08c2b7c3e8200022e"
format_stamp: "Formatted at 2026-08-07 02:22:44 on dist-test-slave-69rg"
server_key: "6ba29b7f36dab556f865f7bdcb6e5a12"
server_key_iv: "88a8b056d65e8263c0b2209d97cc8a09"
server_key_version: "encryptionkey@0"
I20260807 02:22:44.818940 15080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "da55cfa88de049c08c2b7c3e8200022e"
format_stamp: "Formatted at 2026-08-07 02:22:44 on dist-test-slave-69rg"
server_key: "6ba29b7f36dab556f865f7bdcb6e5a12"
server_key_iv: "88a8b056d65e8263c0b2209d97cc8a09"
server_key_version: "encryptionkey@0"
I20260807 02:22:44.823235 15080 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:22:44.826224 15094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:44.827610 15080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 02:22:44.827772 15080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "da55cfa88de049c08c2b7c3e8200022e"
format_stamp: "Formatted at 2026-08-07 02:22:44 on dist-test-slave-69rg"
server_key: "6ba29b7f36dab556f865f7bdcb6e5a12"
server_key_iv: "88a8b056d65e8263c0b2209d97cc8a09"
server_key_version: "encryptionkey@0"
I20260807 02:22:44.827908 15080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:44.832872 15080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:44.833705 15080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:44.833995 15080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:44.834646 15080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:44.835639 15080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:44.835690 15080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:44.835731 15080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:44.835749 15080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:44.845248 15080 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:34053
I20260807 02:22:44.845266 15210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:34053 every 8 connection(s)
I20260807 02:22:44.846428 15080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:22:44.849329 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 15080
I20260807 02:22:44.849474 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:22:44.849871 4981 external_mini_cluster.cc:1428] Setting key 4188b1551cf09f7cd24fdd97e1447038
I20260807 02:22:44.852523 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40139
--builtin_ntp_servers=127.4.221.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 02:22:44.854440 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:44.848526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:38403 (local address 127.4.221.126:40139)
0807 02:22:44.848677 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:44.848681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:44.850222 (+ 1541us) server_negotiation.cc:407] Connection header received
0807 02:22:44.852148 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:44.852152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:44.852219 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:44.852314 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:44.853863 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:44.853869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:44.853872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:44.853890 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:44.853915 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:44.853931 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:44.854045 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:44.854220 (+ 175us) server_negotiation.cc:299] Negotiation successful
0807 02:22:44.854274 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 02:22:44.855607 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:44.848930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40139 (local address 127.4.221.65:38403)
0807 02:22:44.849973 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:44.850043 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:44.851899 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:44.852535 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:44.852543 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:44.852714 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:44.853471 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:22:44.853473 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:44.853523 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:22:44.853615 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:44.854027 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:44.854038 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:44.854045 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 02:22:44.854289 (+ 244us) client_negotiation.cc:240] Negotiation successful
0807 02:22:44.854612 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":186,"threads_started":1}
I20260807 02:22:44.857472 15211 heartbeater.cc:347] Connected to a master server at 127.4.221.126:40139
I20260807 02:22:44.857903 15211 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:44.858618 15211 heartbeater.cc:511] Master 127.4.221.126:40139 requested a full tablet report, sending...
I20260807 02:22:44.861033 15019 ts_manager.cc:194] Registered new tserver with Master: da55cfa88de049c08c2b7c3e8200022e (127.4.221.65:34053)
W20260807 02:22:44.978068 15214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:44.978425 15214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:44.978495 15214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:44.983422 15214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:44.983516 15214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:44.983620 15214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:22:44.988741 15214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:44.990074 15214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:44.991544 15214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:45.001755 15220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:45.001971 15214 server_base.cc:1034] running on GCE node
W20260807 02:22:45.001783 15219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:45.001770 15222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:45.002468 15214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:45.003196 15214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:45.004387 15214 hybrid_clock.cc:648] HybridClock initialized: now 1786069365004355 us; error 47 us; skew 500 ppm
I20260807 02:22:45.006511 15214 webserver.cc:460] Webserver started at http://127.4.221.66:40091/ using document root <none> and password file <none>
I20260807 02:22:45.007113 15214 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:45.007263 15214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:45.007495 15214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:45.009616 15214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "7ea396fe8d4b418bb0741627603b11e2"
format_stamp: "Formatted at 2026-08-07 02:22:45 on dist-test-slave-69rg"
server_key: "09fd36530e0468d834fa78d7a0905868"
server_key_iv: "963549c37de74f43e17dc5e04d0f5d78"
server_key_version: "encryptionkey@0"
I20260807 02:22:45.010267 15214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "7ea396fe8d4b418bb0741627603b11e2"
format_stamp: "Formatted at 2026-08-07 02:22:45 on dist-test-slave-69rg"
server_key: "09fd36530e0468d834fa78d7a0905868"
server_key_iv: "963549c37de74f43e17dc5e04d0f5d78"
server_key_version: "encryptionkey@0"
I20260807 02:22:45.014408 15214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 02:22:45.017184 15228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:45.018486 15214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:22:45.018623 15214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "7ea396fe8d4b418bb0741627603b11e2"
format_stamp: "Formatted at 2026-08-07 02:22:45 on dist-test-slave-69rg"
server_key: "09fd36530e0468d834fa78d7a0905868"
server_key_iv: "963549c37de74f43e17dc5e04d0f5d78"
server_key_version: "encryptionkey@0"
I20260807 02:22:45.018748 15214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:45.023527 15214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:45.024163 15214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:45.024353 15214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:45.024909 15214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:45.025882 15214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:45.025942 15214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:45.026007 15214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:45.026053 15214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:45.035584 15214 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:36485
I20260807 02:22:45.035606 15350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:36485 every 8 connection(s)
I20260807 02:22:45.036580 15214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:22:45.042126 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 15214
I20260807 02:22:45.042321 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:22:45.042665 4981 external_mini_cluster.cc:1428] Setting key 23d71c79242e42f21ed052fd8aba7242
I20260807 02:22:45.043207 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:45.038658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:57725 (local address 127.4.221.126:40139)
0807 02:22:45.038812 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:45.038816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:45.039876 (+ 1060us) server_negotiation.cc:407] Connection header received
0807 02:22:45.041193 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:45.041196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:45.041237 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:45.041331 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:45.042726 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:45.042730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:45.042734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:45.042751 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:45.042779 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:45.042788 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:45.042892 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:45.043016 (+ 124us) server_negotiation.cc:299] Negotiation successful
0807 02:22:45.043066 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 02:22:45.044245 15353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:45.039046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40139 (local address 127.4.221.66:57725)
0807 02:22:45.039728 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:45.039769 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:45.040992 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:45.041529 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:45.041537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:45.041628 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:45.042443 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:22:45.042445 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:45.042488 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:22:45.042560 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:45.042877 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:45.042884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:45.042889 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 02:22:45.043035 (+ 146us) client_negotiation.cc:240] Negotiation successful
0807 02:22:45.043352 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":124,"threads_started":1}
I20260807 02:22:45.044881 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40139
--builtin_ntp_servers=127.4.221.84:39505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260807 02:22:45.045437 15351 heartbeater.cc:347] Connected to a master server at 127.4.221.126:40139
I20260807 02:22:45.045694 15351 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:45.046146 15351 heartbeater.cc:511] Master 127.4.221.126:40139 requested a full tablet report, sending...
I20260807 02:22:45.047247 15019 ts_manager.cc:194] Registered new tserver with Master: 7ea396fe8d4b418bb0741627603b11e2 (127.4.221.66:36485)
W20260807 02:22:45.163656 15354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:45.163991 15354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:45.164036 15354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:45.168813 15354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:45.168898 15354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:45.168977 15354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:22:45.174328 15354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:39505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:45.175621 15354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:45.177170 15354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 02:22:45.187702 15354 server_base.cc:1034] running on GCE node
W20260807 02:22:45.187633 15359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:45.187624 15362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:45.187719 15360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:45.188313 15354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:45.189061 15354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:45.190258 15354 hybrid_clock.cc:648] HybridClock initialized: now 1786069365190237 us; error 39 us; skew 500 ppm
I20260807 02:22:45.192296 15354 webserver.cc:460] Webserver started at http://127.4.221.67:38907/ using document root <none> and password file <none>
I20260807 02:22:45.192924 15354 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:45.193022 15354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:45.193279 15354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:45.195830 15354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "d35ffe6425ca43779838b3e47ce329c8"
format_stamp: "Formatted at 2026-08-07 02:22:45 on dist-test-slave-69rg"
server_key: "83113a16b3c5167225dd563e3354b077"
server_key_iv: "84a44996344bc9fcdaed64c08df1c667"
server_key_version: "encryptionkey@0"
I20260807 02:22:45.196516 15354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "d35ffe6425ca43779838b3e47ce329c8"
format_stamp: "Formatted at 2026-08-07 02:22:45 on dist-test-slave-69rg"
server_key: "83113a16b3c5167225dd563e3354b077"
server_key_iv: "84a44996344bc9fcdaed64c08df1c667"
server_key_version: "encryptionkey@0"
I20260807 02:22:45.200908 15354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 02:22:45.204012 15368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:45.205494 15354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:22:45.205674 15354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "d35ffe6425ca43779838b3e47ce329c8"
format_stamp: "Formatted at 2026-08-07 02:22:45 on dist-test-slave-69rg"
server_key: "83113a16b3c5167225dd563e3354b077"
server_key_iv: "84a44996344bc9fcdaed64c08df1c667"
server_key_version: "encryptionkey@0"
I20260807 02:22:45.205868 15354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:45.211118 15354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:45.211791 15354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:45.211987 15354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:45.212555 15354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:45.213500 15354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:45.213587 15354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:45.213658 15354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:45.213716 15354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:45.223517 15354 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:33849
I20260807 02:22:45.223546 15485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:33849 every 8 connection(s)
I20260807 02:22:45.224535 15354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:22:45.231756 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:45.226552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:46105 (local address 127.4.221.126:40139)
0807 02:22:45.226725 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:45.226730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:45.227960 (+ 1230us) server_negotiation.cc:407] Connection header received
0807 02:22:45.229886 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:45.229891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:45.229953 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:45.230079 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:45.231302 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:45.231305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:45.231308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:22:45.231320 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:45.231341 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:22:45.231349 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:45.231433 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:45.231554 (+ 121us) server_negotiation.cc:299] Negotiation successful
0807 02:22:45.231596 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 02:22:45.232736 15488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:45.227031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40139 (local address 127.4.221.67:46105)
0807 02:22:45.227801 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:45.227845 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:45.229567 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:45.230301 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:45.230308 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:45.230426 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:45.231028 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:22:45.231029 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:45.231070 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:22:45.231140 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:45.231426 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:45.231434 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:45.231438 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 02:22:45.231581 (+ 143us) client_negotiation.cc:240] Negotiation successful
0807 02:22:45.231864 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":121,"threads_started":1}
I20260807 02:22:45.232869 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 15354
I20260807 02:22:45.232983 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:22:45.233289 4981 external_mini_cluster.cc:1428] Setting key a93b103c99ef3c580ff77c14197e9a5d
I20260807 02:22:45.234169 15486 heartbeater.cc:347] Connected to a master server at 127.4.221.126:40139
I20260807 02:22:45.234426 15486 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:45.234897 15486 heartbeater.cc:511] Master 127.4.221.126:40139 requested a full tablet report, sending...
I20260807 02:22:45.235994 15019 ts_manager.cc:194] Registered new tserver with Master: d35ffe6425ca43779838b3e47ce329c8 (127.4.221.67:33849)
I20260807 02:22:45.247853 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260807 02:22:45.251101 15058 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:22:45.250016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.4.221.126:40139)
0807 02:22:45.250166 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:45.250172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:45.250402 (+ 230us) server_negotiation.cc:407] Connection header received
0807 02:22:45.250562 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:45.250565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:45.250605 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:45.250691 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:45.250949 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39660: BlockingRecv error: recv got EOF from 127.0.0.1:39660 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 02:22:45.251199 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 15080
I20260807 02:22:45.258026 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 15214
I20260807 02:22:45.264483 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 15354
I20260807 02:22:45.272337 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 14986
2026-08-07T02:22:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (881 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:22:45 dist-test-slave-69rg krb5kdc[15496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:22:45 dist-test-slave-69rg krb5kdc[15496](info): set up 2 sockets
Aug 07 02:22:45 dist-test-slave-69rg krb5kdc[15496](info): commencing operation
krb5kdc: starting...
W20260807 02:22:46.366685 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:22:46 dist-test-slave-69rg krb5kdc[15496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:22:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:22:46.507157 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:42657
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:42657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:46.622115 15509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:46.622475 15509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:46.622553 15509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:46.626808 15509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:22:46.626924 15509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:46.626964 15509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:46.627000 15509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:22:46.627033 15509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:22:46.632793 15509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:42657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:42657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:46.634186 15509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:46.635327 15509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:46.644462 15514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:46.644462 15515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:46.644596 15517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:46.644907 15509 server_base.cc:1034] running on GCE node
I20260807 02:22:46.645555 15509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:46.646860 15509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:46.648026 15509 hybrid_clock.cc:648] HybridClock initialized: now 1786069366648009 us; error 33 us; skew 500 ppm
Aug 07 02:22:46 dist-test-slave-69rg krb5kdc[15496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069366, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:46.656729 15509 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:22:46.658002 15509 webserver.cc:460] Webserver started at http://127.4.221.126:35075/ using document root <none> and password file <none>
I20260807 02:22:46.658684 15509 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:46.658756 15509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:46.658995 15509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:46.661324 15509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "8c0044d6a52141c5b6fe07e6b5fdabe1"
format_stamp: "Formatted at 2026-08-07 02:22:46 on dist-test-slave-69rg"
server_key: "04ae124d2cd8faef1f8ab3cb92d81e8c"
server_key_iv: "8985304651c1d7e6250e9eef2efea81f"
server_key_version: "encryptionkey@0"
I20260807 02:22:46.662038 15509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "8c0044d6a52141c5b6fe07e6b5fdabe1"
format_stamp: "Formatted at 2026-08-07 02:22:46 on dist-test-slave-69rg"
server_key: "04ae124d2cd8faef1f8ab3cb92d81e8c"
server_key_iv: "8985304651c1d7e6250e9eef2efea81f"
server_key_version: "encryptionkey@0"
I20260807 02:22:46.666349 15509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 02:22:46.668993 15524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:46.670358 15509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:22:46.670513 15509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "8c0044d6a52141c5b6fe07e6b5fdabe1"
format_stamp: "Formatted at 2026-08-07 02:22:46 on dist-test-slave-69rg"
server_key: "04ae124d2cd8faef1f8ab3cb92d81e8c"
server_key_iv: "8985304651c1d7e6250e9eef2efea81f"
server_key_version: "encryptionkey@0"
I20260807 02:22:46.670641 15509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:46.694167 15509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:46.697749 15509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:46.698019 15509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:46.705750 15509 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:42657
I20260807 02:22:46.706089 15577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:42657 every 8 connection(s)
I20260807 02:22:46.706874 15509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:22:46.709867 15578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:46.715090 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 15509
I20260807 02:22:46.715210 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:22:46.715502 4981 external_mini_cluster.cc:1428] Setting key 2e84386706f2d0c535a099e1b8f234a6
I20260807 02:22:46.715827 15578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1: Bootstrap starting.
I20260807 02:22:46.718545 15578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:46.719293 15578 log.cc:824] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:46.721204 15578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1: No bootstrap required, opened a new log
Aug 07 02:22:46 dist-test-slave-69rg krb5kdc[15496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:46.725291 15578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0044d6a52141c5b6fe07e6b5fdabe1" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 42657 } }
I20260807 02:22:46.725520 15578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:46.725575 15578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c0044d6a52141c5b6fe07e6b5fdabe1, State: Initialized, Role: FOLLOWER
I20260807 02:22:46.726089 15578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0044d6a52141c5b6fe07e6b5fdabe1" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 42657 } }
I20260807 02:22:46.726284 15578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:22:46.726364 15578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:22:46.726505 15578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:46.727450 15578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0044d6a52141c5b6fe07e6b5fdabe1" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 42657 } }
I20260807 02:22:46.727794 15578 leader_election.cc:302] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c0044d6a52141c5b6fe07e6b5fdabe1; no voters:
I20260807 02:22:46.728109 15578 leader_election.cc:288] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:22:46.728226 15583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:46.728468 15583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [term 1 LEADER]: Becoming Leader. State: Replica: 8c0044d6a52141c5b6fe07e6b5fdabe1, State: Running, Role: LEADER
I20260807 02:22:46.728808 15583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0044d6a52141c5b6fe07e6b5fdabe1" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 42657 } }
I20260807 02:22:46.729564 15578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:22:46.730474 15584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c0044d6a52141c5b6fe07e6b5fdabe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0044d6a52141c5b6fe07e6b5fdabe1" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 42657 } } }
I20260807 02:22:46.730567 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:46.716908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.4.221.126:42657)
0807 02:22:46.717412 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:46.717428 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:46.717463 (+ 35us) server_negotiation.cc:407] Connection header received
0807 02:22:46.718236 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:46.718255 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:46.718585 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:46.718961 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:46.719781 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:46.720796 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:46.721751 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:46.722281 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:46.722447 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:46.725211 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:46.725216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:46.725229 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:46.725256 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:46.727155 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:46.727596 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:46.727602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:46.727608 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:46.727741 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:46.728045 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:46.728048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:46.728050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:46.728652 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:46.728840 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:46.729155 (+ 315us) server_negotiation.cc:299] Negotiation successful
0807 02:22:46.729471 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":182,"threads_started":1}
I20260807 02:22:46.730609 15584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [sys.catalog]: This master's current role is: LEADER
I20260807 02:22:46.730996 15585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c0044d6a52141c5b6fe07e6b5fdabe1. Latest consensus state: current_term: 1 leader_uuid: "8c0044d6a52141c5b6fe07e6b5fdabe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c0044d6a52141c5b6fe07e6b5fdabe1" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 42657 } } }
I20260807 02:22:46.731107 15585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1 [sys.catalog]: This master's current role is: LEADER
W20260807 02:22:46.734347 15598 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:22:46.734427 15598 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:22:46.734503 15592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:22:46.735248 15592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:22:46.740832 15592 catalog_manager.cc:1347] Generated new cluster ID: 2497d24214a747b8a20e8e17e8fbb993
I20260807 02:22:46.740936 15592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:22:46.763022 15592 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:22:46.764077 15592 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:22:46.769223 15592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1: Generated new TSK 0
I20260807 02:22:46.770059 15592 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:22:46.813241 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:42657
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:46.933418 15605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:46.933751 15605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:46.933801 15605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:46.938023 15605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:46.938102 15605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:46.938181 15605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:22:46.943912 15605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:42657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:46.945065 15605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:46.946144 15605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:46.956346 15614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:46.956489 15605 server_base.cc:1034] running on GCE node
W20260807 02:22:46.956303 15610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:46.956303 15611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:46.957095 15605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:46.957860 15605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:46.959062 15605 hybrid_clock.cc:648] HybridClock initialized: now 1786069366959038 us; error 40 us; skew 500 ppm
Aug 07 02:22:46 dist-test-slave-69rg krb5kdc[15496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069366, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:46.962270 15605 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:22:46.963261 15605 webserver.cc:460] Webserver started at http://127.4.221.65:40523/ using document root <none> and password file <none>
I20260807 02:22:46.963867 15605 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:46.963959 15605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:46.964195 15605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:46.966478 15605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "1aeffaacee1d476d87c71b30974b5978"
format_stamp: "Formatted at 2026-08-07 02:22:46 on dist-test-slave-69rg"
server_key: "6e14bb1f6f14714c067ea30f83607bb8"
server_key_iv: "d290f8db667fdc557a7032ffadf0137b"
server_key_version: "encryptionkey@0"
I20260807 02:22:46.967089 15605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "1aeffaacee1d476d87c71b30974b5978"
format_stamp: "Formatted at 2026-08-07 02:22:46 on dist-test-slave-69rg"
server_key: "6e14bb1f6f14714c067ea30f83607bb8"
server_key_iv: "d290f8db667fdc557a7032ffadf0137b"
server_key_version: "encryptionkey@0"
I20260807 02:22:46.971230 15605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:22:46.974092 15622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:46.975525 15605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:22:46.975669 15605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "1aeffaacee1d476d87c71b30974b5978"
format_stamp: "Formatted at 2026-08-07 02:22:46 on dist-test-slave-69rg"
server_key: "6e14bb1f6f14714c067ea30f83607bb8"
server_key_iv: "d290f8db667fdc557a7032ffadf0137b"
server_key_version: "encryptionkey@0"
I20260807 02:22:46.975798 15605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:46.988679 15605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:46.992058 15605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:46.992265 15605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:46.992869 15605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:46.993698 15605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:46.993765 15605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:46.993870 15605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:46.993916 15605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:47.003290 15605 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:37897
I20260807 02:22:47.003309 15737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:37897 every 8 connection(s)
I20260807 02:22:47.004243 15605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:22:47.011637 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 15605
I20260807 02:22:47.011772 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:22:47.012110 4981 external_mini_cluster.cc:1428] Setting key 443e9135453e5b662c548925a94a5192
Aug 07 02:22:47 dist-test-slave-69rg krb5kdc[15496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069366, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:47.019297 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.006070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:47931 (local address 127.4.221.126:42657)
0807 02:22:47.006230 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:47.006235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:47.007085 (+ 850us) server_negotiation.cc:407] Connection header received
0807 02:22:47.008428 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:47.008431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:47.008480 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:47.008555 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:47.009786 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.010503 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.011967 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.012252 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.012339 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.015445 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:47.015449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:47.015451 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:47.015478 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:47.017027 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:47.017580 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:47.017583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:47.017584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:47.017664 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:47.018081 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:47.018086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:47.018088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:47.018308 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:47.018426 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:47.018981 (+ 555us) server_negotiation.cc:299] Negotiation successful
0807 02:22:47.019121 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:22:47.020193 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.006494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:42657 (local address 127.4.221.65:47931)
0807 02:22:47.006936 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:47.006980 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:47.008234 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:47.008737 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:47.008744 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:47.009138 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:47.009646 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.009660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.010660 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.010663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.011839 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.011851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.012417 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.012422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.012628 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.013280 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:47.013298 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:47.015261 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:47.017153 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:47.017160 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:47.017177 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:47.017464 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:47.017761 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:47.017765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:47.017767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:47.017986 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:47.018428 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:47.018437 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:47.018722 (+ 285us) client_negotiation.cc:769] Sending connection context
0807 02:22:47.018927 (+ 205us) client_negotiation.cc:240] Negotiation successful
0807 02:22:47.019293 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
I20260807 02:22:47.021548 15738 heartbeater.cc:347] Connected to a master server at 127.4.221.126:42657
I20260807 02:22:47.021878 15738 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:47.022573 15738 heartbeater.cc:511] Master 127.4.221.126:42657 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
I20260807 02:22:47.024386 15542 ts_manager.cc:194] Registered new tserver with Master: 1aeffaacee1d476d87c71b30974b5978 (127.4.221.65:37897)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
I20260807 02:22:47.025820 15542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:47931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:22:47.046140 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:42657
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:47.163920 15743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:47.164207 15743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:47.164255 15743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:47.168476 15743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:47.168578 15743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:47.168677 15743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:22:47.174604 15743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:42657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:47.175882 15743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:47.177008 15743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:47.187007 15748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:47.187058 15749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:47.187134 15743 server_base.cc:1034] running on GCE node
W20260807 02:22:47.187417 15752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:47.187872 15743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:47.188494 15743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:47.189668 15743 hybrid_clock.cc:648] HybridClock initialized: now 1786069367189632 us; error 45 us; skew 500 ppm
Aug 07 02:22:47 dist-test-slave-69rg krb5kdc[15496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069367, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:47.192576 15743 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:22:47.193676 15743 webserver.cc:460] Webserver started at http://127.4.221.66:34719/ using document root <none> and password file <none>
I20260807 02:22:47.194327 15743 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:47.194419 15743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:47.194676 15743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:47.196822 15743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "b4837b7862394be28489816d2ac44e55"
format_stamp: "Formatted at 2026-08-07 02:22:47 on dist-test-slave-69rg"
server_key: "dfb199cc13a98dd3b4516a3020926963"
server_key_iv: "3bd6b2c0c3640d1a3ec871c3ad6a3512"
server_key_version: "encryptionkey@0"
I20260807 02:22:47.197428 15743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "b4837b7862394be28489816d2ac44e55"
format_stamp: "Formatted at 2026-08-07 02:22:47 on dist-test-slave-69rg"
server_key: "dfb199cc13a98dd3b4516a3020926963"
server_key_iv: "3bd6b2c0c3640d1a3ec871c3ad6a3512"
server_key_version: "encryptionkey@0"
I20260807 02:22:47.201463 15743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260807 02:22:47.204174 15759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:47.205453 15743 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:22:47.205619 15743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "b4837b7862394be28489816d2ac44e55"
format_stamp: "Formatted at 2026-08-07 02:22:47 on dist-test-slave-69rg"
server_key: "dfb199cc13a98dd3b4516a3020926963"
server_key_iv: "3bd6b2c0c3640d1a3ec871c3ad6a3512"
server_key_version: "encryptionkey@0"
I20260807 02:22:47.205745 15743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:47.225464 15743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:47.229293 15743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:47.229522 15743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:47.230252 15743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:47.231259 15743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:47.231326 15743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:47.231390 15743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:47.231422 15743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:47.240810 15743 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:36277
I20260807 02:22:47.240839 15875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:36277 every 8 connection(s)
I20260807 02:22:47.241995 15743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:22:47.244434 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 15743
I20260807 02:22:47.244552 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:22:47.244899 4981 external_mini_cluster.cc:1428] Setting key f59bb3e63983a7f99e7b401a0ab84349
Aug 07 02:22:47 dist-test-slave-69rg krb5kdc[15496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069367, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
I20260807 02:22:47.258312 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.244059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:60891 (local address 127.4.221.126:42657)
0807 02:22:47.244226 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:47.244231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:47.245324 (+ 1093us) server_negotiation.cc:407] Connection header received
0807 02:22:47.246898 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:47.246904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:47.246968 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:47.247071 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:47.248292 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.248960 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.250494 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.250739 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.250829 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.254019 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:47.254025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:47.254030 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:47.254067 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:47.255944 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:47.256523 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:47.256527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:47.256529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:47.256626 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:47.256976 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:47.256980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:47.256982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:47.257266 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:47.257370 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:47.257990 (+ 620us) server_negotiation.cc:299] Negotiation successful
0807 02:22:47.258124 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 02:22:47.259414 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.244617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:42657 (local address 127.4.221.66:60891)
0807 02:22:47.245163 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:47.245207 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:47.246675 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:47.247226 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:47.247236 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:47.247635 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:47.248139 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.248153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.249104 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.249111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.250369 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.250381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.250910 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.250917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.251157 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.251872 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:47.251899 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:47.253762 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:47.256079 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:47.256084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:47.256100 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:47.256398 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:47.256730 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:47.256732 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:47.256733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:47.256874 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:47.257411 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:47.257416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:47.257706 (+ 290us) client_negotiation.cc:769] Sending connection context
0807 02:22:47.257932 (+ 226us) client_negotiation.cc:240] Negotiation successful
0807 02:22:47.258382 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":154,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
I20260807 02:22:47.260800 15876 heartbeater.cc:347] Connected to a master server at 127.4.221.126:42657
I20260807 02:22:47.261091 15876 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:47.261649 15876 heartbeater.cc:511] Master 127.4.221.126:42657 requested a full tablet report, sending...
I20260807 02:22:47.262843 15542 ts_manager.cc:194] Registered new tserver with Master: b4837b7862394be28489816d2ac44e55 (127.4.221.66:36277)
I20260807 02:22:47.263442 15542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:60891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:22:47.277712 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:42657
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:22:47.396188 15882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:22:47.396497 15882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:22:47.396565 15882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:22:47.400926 15882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:22:47.401016 15882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:22:47.401109 15882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:22:47.406910 15882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:42657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:22:47.408175 15882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:22:47.409348 15882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:22:47.419404 15890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:47.419433 15888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:22:47.419404 15887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:22:47.419749 15882 server_base.cc:1034] running on GCE node
I20260807 02:22:47.420189 15882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:22:47.420913 15882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:22:47.422072 15882 hybrid_clock.cc:648] HybridClock initialized: now 1786069367422058 us; error 34 us; skew 500 ppm
Aug 07 02:22:47 dist-test-slave-69rg krb5kdc[15496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069367, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:22:47.424922 15882 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:22:47.425956 15882 webserver.cc:460] Webserver started at http://127.4.221.67:37483/ using document root <none> and password file <none>
I20260807 02:22:47.426528 15882 fs_manager.cc:359] Metadata directory not provided
I20260807 02:22:47.426590 15882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:22:47.426846 15882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:22:47.429046 15882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "d33e0f024ced4445a85f9f08812a3a15"
format_stamp: "Formatted at 2026-08-07 02:22:47 on dist-test-slave-69rg"
server_key: "e3bd4fa8b642b34c266f1217c5bba523"
server_key_iv: "065a83c67a30982b2d819efcf64d575b"
server_key_version: "encryptionkey@0"
I20260807 02:22:47.429625 15882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "d33e0f024ced4445a85f9f08812a3a15"
format_stamp: "Formatted at 2026-08-07 02:22:47 on dist-test-slave-69rg"
server_key: "e3bd4fa8b642b34c266f1217c5bba523"
server_key_iv: "065a83c67a30982b2d819efcf64d575b"
server_key_version: "encryptionkey@0"
I20260807 02:22:47.433643 15882 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260807 02:22:47.436259 15897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:47.437577 15882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:22:47.437713 15882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "d33e0f024ced4445a85f9f08812a3a15"
format_stamp: "Formatted at 2026-08-07 02:22:47 on dist-test-slave-69rg"
server_key: "e3bd4fa8b642b34c266f1217c5bba523"
server_key_iv: "065a83c67a30982b2d819efcf64d575b"
server_key_version: "encryptionkey@0"
I20260807 02:22:47.437899 15882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:22:47.455468 15882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:22:47.458781 15882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:22:47.459050 15882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:22:47.459689 15882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:22:47.460547 15882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:22:47.460597 15882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:47.460664 15882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:22:47.460700 15882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:22:47.470260 15882 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:41649
I20260807 02:22:47.470279 16015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:41649 every 8 connection(s)
I20260807 02:22:47.471205 15882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:22:47.476147 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 15882
I20260807 02:22:47.476281 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:22:47.476577 4981 external_mini_cluster.cc:1428] Setting key c99765829c6899660c45383def918f09
Aug 07 02:22:47 dist-test-slave-69rg krb5kdc[15496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069367, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:22:47.485842 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.473011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:44535 (local address 127.4.221.126:42657)
0807 02:22:47.473175 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:47.473179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:47.474038 (+ 859us) server_negotiation.cc:407] Connection header received
0807 02:22:47.475464 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:47.475468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:47.475529 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:47.475632 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:47.476864 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.477613 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.479105 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.479404 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.479507 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.482196 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:47.482200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:47.482202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:47.482231 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:47.483661 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:47.484194 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:47.484197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:47.484199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:47.484288 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:47.484636 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:47.484639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:47.484641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:47.484844 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:47.484977 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:47.485505 (+ 528us) server_negotiation.cc:299] Negotiation successful
0807 02:22:47.485641 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:22:47.486732 16018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.473412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:42657 (local address 127.4.221.67:44535)
0807 02:22:47.473902 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:47.473945 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:47.475211 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:47.475806 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:47.475812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:47.476164 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:22:47.476719 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.476741 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.477729 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.477733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.478983 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.478992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.479556 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.479560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.479740 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.480337 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:22:47.480353 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:22:47.482033 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:22:47.483787 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:47.483793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:47.483810 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:47.484087 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:47.484380 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:22:47.484383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:22:47.484385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:22:47.484537 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:22:47.484975 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:22:47.484983 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:22:47.485254 (+ 271us) client_negotiation.cc:769] Sending connection context
0807 02:22:47.485451 (+ 197us) client_negotiation.cc:240] Negotiation successful
0807 02:22:47.485862 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":121,"threads_started":1}
I20260807 02:22:47.487998 16016 heartbeater.cc:347] Connected to a master server at 127.4.221.126:42657
I20260807 02:22:47.488269 16016 heartbeater.cc:464] Registering TS with master...
I20260807 02:22:47.488795 16016 heartbeater.cc:511] Master 127.4.221.126:42657 requested a full tablet report, sending...
I20260807 02:22:47.489920 15541 ts_manager.cc:194] Registered new tserver with Master: d33e0f024ced4445a85f9f08812a3a15 (127.4.221.67:41649)
I20260807 02:22:47.490614 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:22:47.490648 15541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:44535
I20260807 02:22:47.500196 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.492653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.4.221.126:42657)
0807 02:22:47.492798 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:47.492803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:47.492956 (+ 153us) server_negotiation.cc:407] Connection header received
0807 02:22:47.493103 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:47.493106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:47.493153 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:47.493232 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:22:47.493997 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.494518 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.495371 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.495599 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.495703 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.497102 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:22:47.497105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:22:47.497107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:22:47.497137 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:22:47.498440 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:47.498879 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:47.498884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:47.498886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:47.498967 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:22:47.499240 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:22:47.499245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:22:47.499246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:22:47.499494 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:22:47.499634 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:47.499884 (+ 250us) server_negotiation.cc:299] Negotiation successful
0807 02:22:47.500022 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 02:22:47.503587 15541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:22:47.506855 15541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:22:47.521377 16028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.515567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:36277 (local address 127.0.0.1:50178)
0807 02:22:47.515924 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:47.515984 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:47.516125 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:47.516890 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:47.516897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:47.516928 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:47.517050 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.517056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.518320 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.518323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.519584 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.519595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.520826 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.520830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.520932 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.520967 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 02:22:47.521069 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 02:22:47.521211 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":61,"thread_start_us":92,"threads_started":1}
I20260807 02:22:47.521661 16029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.515778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50178 (local address 127.4.221.66:36277)
0807 02:22:47.516167 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:47.516180 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:47.516206 (+ 26us) server_negotiation.cc:407] Connection header received
0807 02:22:47.516289 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:47.516294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:47.516495 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:47.516653 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:47.517182 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.518153 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.519723 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.520523 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.520625 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.520765 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:47.521430 (+ 665us) server_negotiation.cc:299] Negotiation successful
0807 02:22:47.521522 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":166,"threads_started":1}
I20260807 02:22:47.521610 16030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.516293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:37897 (local address 127.0.0.1:41276)
0807 02:22:47.517066 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:47.517083 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:47.517178 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:47.517752 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:47.517755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:47.517774 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:47.517896 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.517902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.519307 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.519310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.520341 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.520350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.521204 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.521207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.521304 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.521337 (+ 33us) client_negotiation.cc:769] Sending connection context
0807 02:22:47.521410 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 02:22:47.521478 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260807 02:22:47.522004 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.516504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41276 (local address 127.4.221.65:37897)
0807 02:22:47.516893 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:47.516907 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:47.517282 (+ 375us) server_negotiation.cc:407] Connection header received
0807 02:22:47.517385 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:47.517393 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:47.517581 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:47.517731 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:47.518013 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.519156 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.520455 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.521088 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.521202 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.521339 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:47.521720 (+ 381us) server_negotiation.cc:299] Negotiation successful
0807 02:22:47.521851 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":86,"threads_started":1}
I20260807 02:22:47.522636 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.516626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:41649 (local address 127.0.0.1:44044)
0807 02:22:47.517349 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:47.517362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:47.517434 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:47.518231 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:47.518237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:47.518263 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:47.518378 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.518385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.519994 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.519997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.521087 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.521095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.522256 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.522263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.522364 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.522398 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 02:22:47.522449 (+ 51us) client_negotiation.cc:240] Negotiation successful
0807 02:22:47.522515 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260807 02:22:47.523324 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.517009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44044 (local address 127.4.221.67:41649)
0807 02:22:47.517692 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:47.517704 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:47.517725 (+ 21us) server_negotiation.cc:407] Connection header received
0807 02:22:47.517861 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:47.517865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:47.518065 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:47.518229 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:47.518495 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.519866 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.521215 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.522080 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.522915 (+ 835us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.523051 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:47.523099 (+ 48us) server_negotiation.cc:299] Negotiation successful
0807 02:22:47.523186 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":106,"threads_started":1}
I20260807 02:22:47.524854 15809 tablet_service.cc:1467] Processing CreateTablet for tablet f4d8c28deffd4d7da8960ceb9ef0ca2e (DEFAULT_TABLE table=test-table [id=28876ea75cae4937b25abd3ff172d55d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:47.524871 15671 tablet_service.cc:1467] Processing CreateTablet for tablet f4d8c28deffd4d7da8960ceb9ef0ca2e (DEFAULT_TABLE table=test-table [id=28876ea75cae4937b25abd3ff172d55d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:47.525933 15809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d8c28deffd4d7da8960ceb9ef0ca2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:47.525935 15671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d8c28deffd4d7da8960ceb9ef0ca2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:47.525743 15946 tablet_service.cc:1467] Processing CreateTablet for tablet f4d8c28deffd4d7da8960ceb9ef0ca2e (DEFAULT_TABLE table=test-table [id=28876ea75cae4937b25abd3ff172d55d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:22:47.526707 15946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d8c28deffd4d7da8960ceb9ef0ca2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:22:47.532369 16034 tablet_bootstrap.cc:492] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55: Bootstrap starting.
I20260807 02:22:47.534027 16035 tablet_bootstrap.cc:492] T f4d8c28deffd4d7da8960ceb9ef0ca2e P 1aeffaacee1d476d87c71b30974b5978: Bootstrap starting.
I20260807 02:22:47.534152 16036 tablet_bootstrap.cc:492] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15: Bootstrap starting.
I20260807 02:22:47.535882 16034 tablet_bootstrap.cc:654] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:47.536281 16035 tablet_bootstrap.cc:654] T f4d8c28deffd4d7da8960ceb9ef0ca2e P 1aeffaacee1d476d87c71b30974b5978: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:47.536396 16036 tablet_bootstrap.cc:654] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15: Neither blocks nor log segments found. Creating new log.
I20260807 02:22:47.536876 16034 log.cc:824] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:47.537081 16035 log.cc:824] T f4d8c28deffd4d7da8960ceb9ef0ca2e P 1aeffaacee1d476d87c71b30974b5978: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:47.537118 16036 log.cc:824] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15: Log is configured to *not* fsync() on all Append() calls
I20260807 02:22:47.539062 16036 tablet_bootstrap.cc:492] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15: No bootstrap required, opened a new log
I20260807 02:22:47.539126 16035 tablet_bootstrap.cc:492] T f4d8c28deffd4d7da8960ceb9ef0ca2e P 1aeffaacee1d476d87c71b30974b5978: No bootstrap required, opened a new log
I20260807 02:22:47.539126 16034 tablet_bootstrap.cc:492] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55: No bootstrap required, opened a new log
I20260807 02:22:47.539240 16036 ts_tablet_manager.cc:1397] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 02:22:47.539304 16034 ts_tablet_manager.cc:1397] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260807 02:22:47.539304 16035 ts_tablet_manager.cc:1397] T f4d8c28deffd4d7da8960ceb9ef0ca2e P 1aeffaacee1d476d87c71b30974b5978: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260807 02:22:47.542882 16036 raft_consensus.cc:348] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4837b7862394be28489816d2ac44e55" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36277 } } peers { permanent_uuid: "1aeffaacee1d476d87c71b30974b5978" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 37897 } } peers { permanent_uuid: "d33e0f024ced4445a85f9f08812a3a15" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41649 } }
I20260807 02:22:47.543154 16036 raft_consensus.cc:374] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:47.543219 16036 raft_consensus.cc:729] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33e0f024ced4445a85f9f08812a3a15, State: Initialized, Role: FOLLOWER
I20260807 02:22:47.543278 16034 raft_consensus.cc:348] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4837b7862394be28489816d2ac44e55" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36277 } } peers { permanent_uuid: "1aeffaacee1d476d87c71b30974b5978" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 37897 } } peers { permanent_uuid: "d33e0f024ced4445a85f9f08812a3a15" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41649 } }
I20260807 02:22:47.543541 16034 raft_consensus.cc:374] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:47.543599 16034 raft_consensus.cc:729] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4837b7862394be28489816d2ac44e55, State: Initialized, Role: FOLLOWER
I20260807 02:22:47.543725 16036 consensus_queue.cc:260] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4837b7862394be28489816d2ac44e55" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36277 } } peers { permanent_uuid: "1aeffaacee1d476d87c71b30974b5978" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 37897 } } peers { permanent_uuid: "d33e0f024ced4445a85f9f08812a3a15" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41649 } }
I20260807 02:22:47.543725 16035 raft_consensus.cc:348] T f4d8c28deffd4d7da8960ceb9ef0ca2e P 1aeffaacee1d476d87c71b30974b5978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4837b7862394be28489816d2ac44e55" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36277 } } peers { permanent_uuid: "1aeffaacee1d476d87c71b30974b5978" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 37897 } } peers { permanent_uuid: "d33e0f024ced4445a85f9f08812a3a15" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41649 } }
I20260807 02:22:47.543958 16035 raft_consensus.cc:374] T f4d8c28deffd4d7da8960ceb9ef0ca2e P 1aeffaacee1d476d87c71b30974b5978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:22:47.544014 16035 raft_consensus.cc:729] T f4d8c28deffd4d7da8960ceb9ef0ca2e P 1aeffaacee1d476d87c71b30974b5978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aeffaacee1d476d87c71b30974b5978, State: Initialized, Role: FOLLOWER
I20260807 02:22:47.544023 16034 consensus_queue.cc:260] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4837b7862394be28489816d2ac44e55" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36277 } } peers { permanent_uuid: "1aeffaacee1d476d87c71b30974b5978" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 37897 } } peers { permanent_uuid: "d33e0f024ced4445a85f9f08812a3a15" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41649 } }
I20260807 02:22:47.544428 16035 consensus_queue.cc:260] T f4d8c28deffd4d7da8960ceb9ef0ca2e P 1aeffaacee1d476d87c71b30974b5978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4837b7862394be28489816d2ac44e55" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36277 } } peers { permanent_uuid: "1aeffaacee1d476d87c71b30974b5978" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 37897 } } peers { permanent_uuid: "d33e0f024ced4445a85f9f08812a3a15" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41649 } }
I20260807 02:22:47.544875 15876 heartbeater.cc:503] Master 127.4.221.126:42657 was elected leader, sending a full tablet report...
I20260807 02:22:47.544935 16036 ts_tablet_manager.cc:1428] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 02:22:47.545043 16034 ts_tablet_manager.cc:1428] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 02:22:47.545253 16016 heartbeater.cc:503] Master 127.4.221.126:42657 was elected leader, sending a full tablet report...
I20260807 02:22:47.545297 16035 ts_tablet_manager.cc:1428] T f4d8c28deffd4d7da8960ceb9ef0ca2e P 1aeffaacee1d476d87c71b30974b5978: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 02:22:47.545608 15738 heartbeater.cc:503] Master 127.4.221.126:42657 was elected leader, sending a full tablet report...
I20260807 02:22:47.681849 16041 raft_consensus.cc:482] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:22:47.682089 16041 raft_consensus.cc:504] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4837b7862394be28489816d2ac44e55" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36277 } } peers { permanent_uuid: "1aeffaacee1d476d87c71b30974b5978" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 37897 } } peers { permanent_uuid: "d33e0f024ced4445a85f9f08812a3a15" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41649 } }
I20260807 02:22:47.683226 16041 leader_election.cc:288] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1aeffaacee1d476d87c71b30974b5978 (127.4.221.65:37897), d33e0f024ced4445a85f9f08812a3a15 (127.4.221.67:41649)
I20260807 02:22:47.688409 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.684100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:40957 (local address 127.4.221.67:41649)
0807 02:22:47.684246 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:47.684252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:47.684418 (+ 166us) server_negotiation.cc:407] Connection header received
0807 02:22:47.684522 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:47.684524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:47.684590 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:47.684673 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:47.684918 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.685597 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.686993 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.687632 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.687777 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.687870 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:47.688215 (+ 345us) server_negotiation.cc:299] Negotiation successful
0807 02:22:47.688286 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 02:22:47.688409 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.683700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:42757 (local address 127.4.221.65:37897)
0807 02:22:47.683867 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:47.683872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:47.683889 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:22:47.684084 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:47.684087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:47.684146 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:47.684266 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:22:47.684597 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.685402 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.686978 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.687637 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.687777 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.687853 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:47.688169 (+ 316us) server_negotiation.cc:299] Negotiation successful
0807 02:22:47.688253 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 02:22:47.688467 15879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.683595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:37897 (local address 127.4.221.66:42757)
0807 02:22:47.683767 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:47.683786 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:47.683909 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:47.684303 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:47.684307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:47.684333 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:47.684476 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.684486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.685517 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.685519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.686827 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.686839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.687812 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.687816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.687989 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.688037 (+ 48us) client_negotiation.cc:769] Sending connection context
0807 02:22:47.688139 (+ 102us) client_negotiation.cc:240] Negotiation successful
0807 02:22:47.688263 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":73}
I20260807 02:22:47.688668 16044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.683925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:41649 (local address 127.4.221.66:40957)
0807 02:22:47.684315 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:47.684329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:22:47.684425 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:22:47.684695 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:22:47.684697 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:22:47.684714 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:22:47.684808 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.684813 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.685696 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.685698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.686878 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:22:47.686889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.687812 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.687816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:22:47.687984 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.688044 (+ 60us) client_negotiation.cc:769] Sending connection context
0807 02:22:47.688190 (+ 146us) client_negotiation.cc:240] Negotiation successful
0807 02:22:47.688263 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":32,"thread_start_us":150,"threads_started":1}
I20260807 02:22:47.689244 15691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d8c28deffd4d7da8960ceb9ef0ca2e" candidate_uuid: "b4837b7862394be28489816d2ac44e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aeffaacee1d476d87c71b30974b5978" is_pre_election: true
I20260807 02:22:47.689321 15969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d8c28deffd4d7da8960ceb9ef0ca2e" candidate_uuid: "b4837b7862394be28489816d2ac44e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33e0f024ced4445a85f9f08812a3a15" is_pre_election: true
I20260807 02:22:47.689591 15691 raft_consensus.cc:2454] T f4d8c28deffd4d7da8960ceb9ef0ca2e P 1aeffaacee1d476d87c71b30974b5978 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4837b7862394be28489816d2ac44e55 in term 0.
I20260807 02:22:47.689646 15969 raft_consensus.cc:2454] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4837b7862394be28489816d2ac44e55 in term 0.
I20260807 02:22:47.690155 15761 leader_election.cc:302] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aeffaacee1d476d87c71b30974b5978, b4837b7862394be28489816d2ac44e55; no voters:
I20260807 02:22:47.690466 16041 raft_consensus.cc:2781] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:22:47.690572 16041 raft_consensus.cc:482] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:22:47.690649 16041 raft_consensus.cc:3037] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:47.691931 16041 raft_consensus.cc:504] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4837b7862394be28489816d2ac44e55" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36277 } } peers { permanent_uuid: "1aeffaacee1d476d87c71b30974b5978" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 37897 } } peers { permanent_uuid: "d33e0f024ced4445a85f9f08812a3a15" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41649 } }
I20260807 02:22:47.692337 16041 leader_election.cc:288] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [CANDIDATE]: Term 1 election: Requested vote from peers 1aeffaacee1d476d87c71b30974b5978 (127.4.221.65:37897), d33e0f024ced4445a85f9f08812a3a15 (127.4.221.67:41649)
I20260807 02:22:47.692776 15691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d8c28deffd4d7da8960ceb9ef0ca2e" candidate_uuid: "b4837b7862394be28489816d2ac44e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aeffaacee1d476d87c71b30974b5978"
I20260807 02:22:47.692816 15969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d8c28deffd4d7da8960ceb9ef0ca2e" candidate_uuid: "b4837b7862394be28489816d2ac44e55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33e0f024ced4445a85f9f08812a3a15"
I20260807 02:22:47.692934 15691 raft_consensus.cc:3037] T f4d8c28deffd4d7da8960ceb9ef0ca2e P 1aeffaacee1d476d87c71b30974b5978 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:47.692952 15969 raft_consensus.cc:3037] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:22:47.694144 15969 raft_consensus.cc:2454] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4837b7862394be28489816d2ac44e55 in term 1.
I20260807 02:22:47.694239 15691 raft_consensus.cc:2454] T f4d8c28deffd4d7da8960ceb9ef0ca2e P 1aeffaacee1d476d87c71b30974b5978 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4837b7862394be28489816d2ac44e55 in term 1.
I20260807 02:22:47.694537 15761 leader_election.cc:302] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4837b7862394be28489816d2ac44e55, d33e0f024ced4445a85f9f08812a3a15; no voters:
I20260807 02:22:47.694756 16041 raft_consensus.cc:2781] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:22:47.695013 16041 raft_consensus.cc:686] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [term 1 LEADER]: Becoming Leader. State: Replica: b4837b7862394be28489816d2ac44e55, State: Running, Role: LEADER
I20260807 02:22:47.695367 16041 consensus_queue.cc:237] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4837b7862394be28489816d2ac44e55" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36277 } } peers { permanent_uuid: "1aeffaacee1d476d87c71b30974b5978" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 37897 } } peers { permanent_uuid: "d33e0f024ced4445a85f9f08812a3a15" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41649 } }
I20260807 02:22:47.698714 15541 catalog_manager.cc:5495] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4837b7862394be28489816d2ac44e55 (127.4.221.66). New cstate: current_term: 1 leader_uuid: "b4837b7862394be28489816d2ac44e55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4837b7862394be28489816d2ac44e55" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 36277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1aeffaacee1d476d87c71b30974b5978" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33e0f024ced4445a85f9f08812a3a15" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41649 } health_report { overall_health: UNKNOWN } } }
I20260807 02:22:47.717720 16029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.713483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.4.221.66:36277)
0807 02:22:47.713609 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0807 02:22:47.713613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:22:47.713626 (+ 13us) server_negotiation.cc:407] Connection header received
0807 02:22:47.713801 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:22:47.713804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:22:47.713908 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:22:47.714003 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:22:47.714268 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.714782 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.716044 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:22:47.716317 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:22:47.716416 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:22:47.716803 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:22:47.717250 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:22:47.717353 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 02:22:47.717458 (+ 105us) server_negotiation.cc:299] Negotiation successful
0807 02:22:47.717516 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260807 02:22:47.722113 16017 tablet.cc:2369] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:22:47.724526 15969 raft_consensus.cc:1261] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15 [term 1 FOLLOWER]: Refusing update from remote peer b4837b7862394be28489816d2ac44e55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:47.724527 15691 raft_consensus.cc:1261] T f4d8c28deffd4d7da8960ceb9ef0ca2e P 1aeffaacee1d476d87c71b30974b5978 [term 1 FOLLOWER]: Refusing update from remote peer b4837b7862394be28489816d2ac44e55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:22:47.725293 16045 consensus_queue.cc:1035] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aeffaacee1d476d87c71b30974b5978" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:47.725603 16041 consensus_queue.cc:1035] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33e0f024ced4445a85f9f08812a3a15" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:22:47.734700 16049 mvcc.cc:205] Tried to move back new op lower bound from 7315740130191785984 to 7315740130083041280. Current Snapshot: MvccSnapshot[applied={T|T < 7315740130191785984}]
I20260807 02:22:47.734946 16050 mvcc.cc:205] Tried to move back new op lower bound from 7315740130191785984 to 7315740130083041280. Current Snapshot: MvccSnapshot[applied={T|T < 7315740130191785984}]
I20260807 02:22:47.742056 15541 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:47.742262 15541 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:22:47.744899 15541 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c0044d6a52141c5b6fe07e6b5fdabe1: Sending DeleteTablet for 3 replicas of tablet f4d8c28deffd4d7da8960ceb9ef0ca2e
I20260807 02:22:47.745685 15809 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d8c28deffd4d7da8960ceb9ef0ca2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:47 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:50178
I20260807 02:22:47.746066 15671 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d8c28deffd4d7da8960ceb9ef0ca2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:47 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:41276
I20260807 02:22:47.746279 16058 tablet_replica.cc:330] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55: stopping tablet replica
I20260807 02:22:47.746280 15946 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d8c28deffd4d7da8960ceb9ef0ca2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:22:47 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:44044
I20260807 02:22:47.746611 16058 raft_consensus.cc:2229] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:22:47.746920 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 15605
I20260807 02:22:47.747007 16058 raft_consensus.cc:2258] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:47.749001 16058 ts_tablet_manager.cc:1905] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:47.751080 16060 tablet_replica.cc:330] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15: stopping tablet replica
I20260807 02:22:47.751488 16060 raft_consensus.cc:2229] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:22:47.751927 16060 raft_consensus.cc:2258] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:22:47.752254 16058 ts_tablet_manager.cc:1918] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:47.752362 16058 log.cc:1196] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/f4d8c28deffd4d7da8960ceb9ef0ca2e
I20260807 02:22:47.752731 16058 ts_tablet_manager.cc:1939] T f4d8c28deffd4d7da8960ceb9ef0ca2e P b4837b7862394be28489816d2ac44e55: Deleting consensus metadata
I20260807 02:22:47.753887 16060 ts_tablet_manager.cc:1905] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:22:47.754026 15525 catalog_manager.cc:4841] TS b4837b7862394be28489816d2ac44e55 (127.4.221.66:36277): tablet f4d8c28deffd4d7da8960ceb9ef0ca2e (table test-table [id=28876ea75cae4937b25abd3ff172d55d]) successfully deleted
I20260807 02:22:47.756130 16060 ts_tablet_manager.cc:1918] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:22:47.756244 16060 log.cc:1196] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/f4d8c28deffd4d7da8960ceb9ef0ca2e
I20260807 02:22:47.756542 16060 ts_tablet_manager.cc:1939] T f4d8c28deffd4d7da8960ceb9ef0ca2e P d33e0f024ced4445a85f9f08812a3a15: Deleting consensus metadata
W20260807 02:22:47.757040 15525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:37897 (error 108)
I20260807 02:22:47.757758 15525 catalog_manager.cc:4841] TS d33e0f024ced4445a85f9f08812a3a15 (127.4.221.67:41649): tablet f4d8c28deffd4d7da8960ceb9ef0ca2e (table test-table [id=28876ea75cae4937b25abd3ff172d55d]) successfully deleted
I20260807 02:22:47.757983 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 15743
I20260807 02:22:47.759205 16028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:22:47.758253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:37897 (local address 127.0.0.1:41288)
0807 02:22:47.759046 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0807 02:22:47.759116 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:37897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":711}
W20260807 02:22:47.759989 15525 catalog_manager.cc:4584] TS 1aeffaacee1d476d87c71b30974b5978 (127.4.221.65:37897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4d8c28deffd4d7da8960ceb9ef0ca2e: Network error: Client connection negotiation failed: client connection to 127.4.221.65:37897: connect: Connection refused (error 111)
I20260807 02:22:47.764386 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 15882
I20260807 02:22:47.769784 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 15509
2026-08-07T02:22:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2499 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:22:47 dist-test-slave-69rg krb5kdc[16064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:22:47 dist-test-slave-69rg krb5kdc[16064](info): set up 2 sockets
Aug 07 02:22:47 dist-test-slave-69rg krb5kdc[16064](info): commencing operation
krb5kdc: starting...
W20260807 02:22:48.839264 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:22:48 dist-test-slave-69rg krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:22:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:22:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/rangeradmin_127.4.221.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/rangeradmin_127.4.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/rangerlookup_127.4.221.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/rangerlookup_127.4.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/HTTP_127.4.221.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/HTTP_127.4.221.84@KRBTEST.COM.keytab.
I20260807 02:22:49.033342 4981 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBkVddJ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/postgres -l logfile start
2026-08-07 02:22:51.826 UTC [16096] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 02:22:51.826 UTC [16096] LOG: listening on IPv4 address "127.4.221.84", port 42167
2026-08-07 02:22:51.830 UTC [16096] LOG: listening on Unix socket "/tmp/.s.PGSQL.42167"
2026-08-07 02:22:51.856 UTC [16099] LOG: database system was shut down at 2026-08-07 02:22:50 UTC
2026-08-07 02:22:51.861 UTC [16096] LOG: database system is ready to accept connections
I20260807 02:22:52.838857 4981 mini_postgres.cc:96] Postgres bound to 42167
2026-08-07 02:22:52.848 UTC [16107] FATAL: database "slave" does not exist
127.4.221.84:42167 - accepting connections
I20260807 02:22:52.849607 4981 mini_ranger.cc:161] Starting Ranger...
I20260807 02:22:52.868593 4981 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 02:22:53.393666 4981 mini_ranger.cc:87] Created ranger Postgres database
I20260807 02:22:53.393774 4981 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-admin
2026-08-07 02:22:53,754 [I] DB FLAVOR :POSTGRES
2026-08-07 02:22:53,754 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 02:22:53,755 [I] Checking connection..
2026-08-07 02:22:53,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 02:22:54,021 [I] Checking connection passed.
2026-08-07 02:22:54,021 [I] --------- Verifying version history table ---------
2026-08-07 02:22:54,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 02:22:54,265 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 02:22:54,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 02:22:54,525 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 02:22:54,525 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 02:22:54,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 02:22:54.775 UTC [16173] WARNING: there is no transaction in progress
2026-08-07 02:22:54,782 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 02:22:54,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 02:22:55,026 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 02:22:55,026 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 02:22:55,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 02:22:55,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 02:22:55,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-69rg.c.gcp-upstream.internal','N') ;"
2026-08-07 02:22:55,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 02:22:56,009 [I] Table x_portal_user does not exist in database ranger
2026-08-07 02:22:56,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 02:22:56,273 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 02:22:56,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 02:22:56,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 02:22:56,771 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 02:22:56,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 02:22:57.798 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:57.815 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:57.834 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:57.851 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:57.870 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:57.891 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:57.955 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:57.964 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:57.980 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:57.991 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:58.002 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:58.020 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:58.032 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:58.040 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:58.048 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:58.057 UTC [16286] WARNING: there is no transaction in progress
2026-08-07 02:22:58,788 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 02:22:58,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 02:22:59,057 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 02:22:59,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 02:22:59,333 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 02:22:59,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 02:22:59,608 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 02:22:59,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 02:22:59,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 02:23:00,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-69rg.c.gcp-upstream.internal';"
2026-08-07 02:23:00,351 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 02:23:00,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 02:23:00,600 [I] DB_PATCHES have already been applied
I20260807 02:23:00.606194 4981 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBkVddJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 02:23:00.606277 4981 mini_ranger.cc:193] Using host: 127.4.221.84
I20260807 02:23:00.611018 4981 mini_ranger.cc:227] Ranger admin URL: http://127.4.221.84:43245
Aug 07, 2026 2:23:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 2:23:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 2:23:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 2:23:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 2:23:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 2:23:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 2:23:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 02:23:01.760774 4981 mini_ranger.cc:160] Time spent starting Ranger: real 8.911s user 0.005s sys 0.009s
Aug 07, 2026 2:23:23 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 2:23:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 07, 2026 2:23:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 2:23:24 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 2:23:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 2:23:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 2:23:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 2:23:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 2:23:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 2:23:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 2:23:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 2:23:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 2:23:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 2:23:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 2:23:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 2:23:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 2:23:24 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 2:23:24 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 02:23:25.618520 4981 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/rangerkms_127.4.221.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/rangerkms_127.4.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.221.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.221.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.221.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/HTTP_127.4.221.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.221.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/HTTP_127.4.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 02:23:25 dist-test-slave-69rg krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069405, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 02:23:25.721490 4981 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 02:23:25.733718 4981 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 02:23:26.807111 4981 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 02:23:26.807260 4981 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-kms
I20260807 02:23:26.807417 4981 mini_ranger_kms.cc:227] Using postgres at 127.4.221.84:42167
2026-08-07 02:23:27,099 [I] DB FLAVOR :POSTGRES
2026-08-07 02:23:27,100 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 02:23:27,100 [I] Checking connection
2026-08-07 02:23:27,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 02:23:27,400 [I] connection success
2026-08-07 02:23:27,400 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 02:23:27,400 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 02:23:27,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 02:23:27,699 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 02:23:27,699 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 02:23:27,700 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 02:23:27,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:42167/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 02:23:27,989 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 02:23:28.542531 4981 mini_ranger_kms.cc:310] Created kms service
I20260807 02:23:28.734784 4981 mini_ranger_kms.cc:326] Created kudu user
I20260807 02:23:28.782883 4981 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 02:23:28 dist-test-slave-69rg krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069405, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:23:29.226449 4981 mini_ranger_kms.cc:384] Added ranger policy
I20260807 02:23:29.226620 4981 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 02:23:29.226648 4981 mini_ranger_kms.cc:240] Using host: 127.4.221.84
I20260807 02:23:29.231688 4981 mini_ranger_kms.cc:276] Ranger KMS PID: 16474
I20260807 02:23:29.231853 4981 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.221.84:38975
Aug 07, 2026 2:23:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 2:23:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 2:23:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 2:23:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 2:23:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 2:23:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260807 02:23:31.544446 4981 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.823s user 0.015s sys 0.252s
I20260807 02:23:31.544600 4981 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 02:23:31.544728 4981 mini_ranger_kms.cc:401] 127.4.221.84:38975/kms/v1/keys
Aug 07 02:23:31 dist-test-slave-69rg krb5kdc[16064](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786069411, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.221.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 07 02:23:31 dist-test-slave-69rg krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069411, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.221.84@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
Aug 07 02:23:32 dist-test-slave-69rg krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069411, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.221.84@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
Aug 07 02:23:34 dist-test-slave-69rg krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:23:34.767083 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:39495
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:39495
--ranger_config_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.221.84:38975/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:23:34.882467 16525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:23:34.882750 16525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:23:34.882792 16525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:23:34.887256 16525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:23:34.887339 16525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:23:34.887360 16525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:23:34.887377 16525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:23:34.887392 16525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:23:34.893262 16525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.221.84:38975/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:39495
--ranger_config_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:39495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:23:34.894469 16525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:23:34.895468 16525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:23:34.904795 16533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:23:34.904847 16525 server_base.cc:1034] running on GCE node
W20260807 02:23:34.904861 16531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:23:34.904819 16530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:23:34.905649 16525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:23:34.906894 16525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:23:34.908093 16525 hybrid_clock.cc:648] HybridClock initialized: now 1786069414908070 us; error 44 us; skew 500 ppm
Aug 07 02:23:34 dist-test-slave-69rg krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069414, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:23:34.911234 16525 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:23:34.912175 16525 webserver.cc:460] Webserver started at http://127.4.221.126:38919/ using document root <none> and password file <none>
I20260807 02:23:34.912742 16525 fs_manager.cc:359] Metadata directory not provided
I20260807 02:23:34.912827 16525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:23:34.913057 16525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:23:34 dist-test-slave-69rg krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069414, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:23:34.971264 16525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "74bc7c1d9cac444195fa12112de5e170"
format_stamp: "Formatted at 2026-08-07 02:23:34 on dist-test-slave-69rg"
server_key: "cbd29a7de7d93e3b8570815779c522df"
server_key_iv: "b48ac92bda43d816e5c34d2159604234"
server_key_version: "kuduclusterkey@0"
I20260807 02:23:34.972007 16525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "74bc7c1d9cac444195fa12112de5e170"
format_stamp: "Formatted at 2026-08-07 02:23:34 on dist-test-slave-69rg"
server_key: "cbd29a7de7d93e3b8570815779c522df"
server_key_iv: "b48ac92bda43d816e5c34d2159604234"
server_key_version: "kuduclusterkey@0"
I20260807 02:23:34.977197 16525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260807 02:23:34.992183 16541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:23:34.993680 16525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:23:34.993856 16525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "74bc7c1d9cac444195fa12112de5e170"
format_stamp: "Formatted at 2026-08-07 02:23:34 on dist-test-slave-69rg"
server_key: "cbd29a7de7d93e3b8570815779c522df"
server_key_iv: "b48ac92bda43d816e5c34d2159604234"
server_key_version: "kuduclusterkey@0"
I20260807 02:23:34.994027 16525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:23:35.013013 16525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:23:35.016747 16525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:23:35.016995 16525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:23:35.025807 16525 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:39495
I20260807 02:23:35.025843 16593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:39495 every 8 connection(s)
I20260807 02:23:35.026898 16525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:23:35.030572 16594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:23:35.036669 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 16525
I20260807 02:23:35.036849 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:23:35.037168 4981 external_mini_cluster.cc:1428] Setting key e1f8b057cdf31411af5aab7d53ef08f5
I20260807 02:23:35.038385 16594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170: Bootstrap starting.
I20260807 02:23:35.041586 16594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170: Neither blocks nor log segments found. Creating new log.
I20260807 02:23:35.042563 16594 log.cc:824] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170: Log is configured to *not* fsync() on all Append() calls
I20260807 02:23:35.045506 16594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170: No bootstrap required, opened a new log
Aug 07 02:23:35 dist-test-slave-69rg krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:23:35.050300 16594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74bc7c1d9cac444195fa12112de5e170" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39495 } }
I20260807 02:23:35.050745 16594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:23:35.050938 16594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bc7c1d9cac444195fa12112de5e170, State: Initialized, Role: FOLLOWER
I20260807 02:23:35.051595 16594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74bc7c1d9cac444195fa12112de5e170" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39495 } }
I20260807 02:23:35.051795 16594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:23:35.051891 16594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:23:35.052004 16594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:23:35.052291 16597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:35.038858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.4.221.126:39495)
0807 02:23:35.039329 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0807 02:23:35.039346 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 02:23:35.039382 (+ 36us) server_negotiation.cc:407] Connection header received
0807 02:23:35.040045 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:23:35.040060 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:23:35.040416 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:23:35.040768 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:23:35.041615 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:35.042796 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:35.043910 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:35.044407 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:35.044579 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:35.047410 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:23:35.047416 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:23:35.047431 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:23:35.047458 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:23:35.049150 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:23:35.049622 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:23:35.049627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:23:35.049631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:23:35.049745 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:23:35.050060 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:23:35.050064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:23:35.050066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:23:35.050493 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:23:35.050738 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0807 02:23:35.050968 (+ 230us) server_negotiation.cc:299] Negotiation successful
0807 02:23:35.051349 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":161,"threads_started":1}
I20260807 02:23:35.053283 16594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74bc7c1d9cac444195fa12112de5e170" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39495 } }
I20260807 02:23:35.053716 16594 leader_election.cc:302] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74bc7c1d9cac444195fa12112de5e170; no voters:
I20260807 02:23:35.054131 16594 leader_election.cc:288] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:23:35.054237 16599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:23:35.054484 16599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [term 1 LEADER]: Becoming Leader. State: Replica: 74bc7c1d9cac444195fa12112de5e170, State: Running, Role: LEADER
I20260807 02:23:35.054831 16599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74bc7c1d9cac444195fa12112de5e170" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39495 } }
I20260807 02:23:35.055544 16594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:23:35.056095 16601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74bc7c1d9cac444195fa12112de5e170. Latest consensus state: current_term: 1 leader_uuid: "74bc7c1d9cac444195fa12112de5e170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74bc7c1d9cac444195fa12112de5e170" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39495 } } }
I20260807 02:23:35.056216 16601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [sys.catalog]: This master's current role is: LEADER
I20260807 02:23:35.056922 16600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74bc7c1d9cac444195fa12112de5e170" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74bc7c1d9cac444195fa12112de5e170" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39495 } } }
I20260807 02:23:35.057009 16600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170 [sys.catalog]: This master's current role is: LEADER
I20260807 02:23:35.057859 16594 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 02:23:35.058032 16604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:23:35.064774 16604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:23:35.080536 16604 catalog_manager.cc:1347] Generated new cluster ID: be20cf6e1dfa49e899335918cb2125e0
I20260807 02:23:35.080746 16604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:23:35.105535 16604 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:23:35.107144 16604 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:23:35.126665 16604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170: Generated new TSK 0
I20260807 02:23:35.127753 16604 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:23:35.833345 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.221.84:38975/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:39495
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:23:35.976894 16640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:23:35.977393 16640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:23:35.977509 16640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:23:35.981982 16640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:23:35.982036 16640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:23:35.982125 16640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:23:35.988101 16640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.221.84:38975/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:39495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:23:35.989390 16640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:23:35.990641 16640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:23:36.000941 16649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:23:36.001153 16651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:23:36.001089 16648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:23:36.002501 16640 server_base.cc:1034] running on GCE node
I20260807 02:23:36.003034 16640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:23:36.003927 16640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:23:36.005116 16640 hybrid_clock.cc:648] HybridClock initialized: now 1786069416005066 us; error 66 us; skew 500 ppm
Aug 07 02:23:36 dist-test-slave-69rg krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:23:36.008170 16640 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:23:36.009234 16640 webserver.cc:460] Webserver started at http://127.4.221.65:37329/ using document root <none> and password file <none>
I20260807 02:23:36.009889 16640 fs_manager.cc:359] Metadata directory not provided
I20260807 02:23:36.009984 16640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:23:36.010236 16640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:23:36 dist-test-slave-69rg krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:23:36.024356 16640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "2b0982a754cd4baab28266c18f1315e2"
format_stamp: "Formatted at 2026-08-07 02:23:36 on dist-test-slave-69rg"
server_key: "8fa5835863425d639ad0a4570bf3b044"
server_key_iv: "c8083c82733c95480fd2f1d28421a189"
server_key_version: "kuduclusterkey@0"
I20260807 02:23:36.025413 16640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "2b0982a754cd4baab28266c18f1315e2"
format_stamp: "Formatted at 2026-08-07 02:23:36 on dist-test-slave-69rg"
server_key: "8fa5835863425d639ad0a4570bf3b044"
server_key_iv: "c8083c82733c95480fd2f1d28421a189"
server_key_version: "kuduclusterkey@0"
I20260807 02:23:36.030791 16640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260807 02:23:36.067173 16658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:23:36.069141 16640 fs_manager.cc:725] Time spent opening block manager: real 0.023s user 0.001s sys 0.000s
I20260807 02:23:36.069293 16640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "2b0982a754cd4baab28266c18f1315e2"
format_stamp: "Formatted at 2026-08-07 02:23:36 on dist-test-slave-69rg"
server_key: "8fa5835863425d639ad0a4570bf3b044"
server_key_iv: "c8083c82733c95480fd2f1d28421a189"
server_key_version: "kuduclusterkey@0"
I20260807 02:23:36.069442 16640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:23:36.097410 16640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:23:36.101579 16640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:23:36.101907 16640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:23:36.102933 16640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:23:36.104120 16640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:23:36.104189 16640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:23:36.104247 16640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:23:36.104305 16640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260807 02:23:36.115535 16640 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:39671
I20260807 02:23:36.116362 16779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:39671 every 8 connection(s)
I20260807 02:23:36.116822 16640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:23:36.123418 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 16640
I20260807 02:23:36.123565 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:23:36.123932 4981 external_mini_cluster.cc:1428] Setting key a58fa97249687749b0fa8e7d21d99a6e
Aug 07 02:23:36 dist-test-slave-69rg krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:23:36.137688 16783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:36.119834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:39495 (local address 127.4.221.65:55211)
0807 02:23:36.120857 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0807 02:23:36.120914 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 02:23:36.122000 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:23:36.122662 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:23:36.122672 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:23:36.123038 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:23:36.123724 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:36.123745 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.125184 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.125188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:36.126707 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:36.126832 (+ 125us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.128033 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.128040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:36.128382 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:36.129247 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:23:36.129295 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:23:36.131533 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:23:36.134122 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:23:36.134130 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:23:36.134149 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:23:36.134543 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:23:36.134888 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:23:36.134891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:23:36.134893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:23:36.135101 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:23:36.135613 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:23:36.135620 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:23:36.135980 (+ 360us) client_negotiation.cc:769] Sending connection context
0807 02:23:36.136212 (+ 232us) client_negotiation.cc:240] Negotiation successful
0807 02:23:36.136569 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":149,"threads_started":1}
I20260807 02:23:36.138521 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:36.119350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:55211 (local address 127.4.221.126:39495)
0807 02:23:36.120946 (+ 1596us) server_negotiation.cc:206] Beginning negotiation
0807 02:23:36.120956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:23:36.121006 (+ 50us) server_negotiation.cc:407] Connection header received
0807 02:23:36.122254 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:23:36.122258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:23:36.122322 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:23:36.122442 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:23:36.123923 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.125027 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.127033 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.127557 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.127707 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:36.131768 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:23:36.131772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:23:36.131775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:23:36.131816 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:23:36.133936 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:23:36.134670 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:23:36.134674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:23:36.134676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:23:36.134771 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:23:36.135213 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:23:36.135216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:23:36.135218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:23:36.135465 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:23:36.135559 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 02:23:36.138210 (+ 2651us) server_negotiation.cc:299] Negotiation successful
0807 02:23:36.138355 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1511,"thread_start_us":100,"threads_started":1}
I20260807 02:23:36.139802 16780 heartbeater.cc:347] Connected to a master server at 127.4.221.126:39495
I20260807 02:23:36.140053 16780 heartbeater.cc:464] Registering TS with master...
I20260807 02:23:36.140652 16780 heartbeater.cc:511] Master 127.4.221.126:39495 requested a full tablet report, sending...
I20260807 02:23:36.142529 16558 ts_manager.cc:194] Registered new tserver with Master: 2b0982a754cd4baab28266c18f1315e2 (127.4.221.65:39671)
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
I20260807 02:23:36.144214 16558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:55211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:23:36.167106 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.221.84:38975/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:39495
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:23:36 dist-test-slave-69rg krb5kdc[16064](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786069416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260807 02:23:36.337949 16786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:23:36.338395 16786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:23:36.338472 16786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:23:36.344923 16786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:23:36.345019 16786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:23:36.345113 16786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:23:36.353629 16786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.221.84:38975/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:39495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:23:36.355237 16786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:23:36.356771 16786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:23:36.370609 16794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:23:36.370735 16796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:23:36.370826 16792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:23:36.371551 16786 server_base.cc:1034] running on GCE node
I20260807 02:23:36.372056 16786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:23:36.372885 16786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:23:36.374078 16786 hybrid_clock.cc:648] HybridClock initialized: now 1786069416374064 us; error 36 us; skew 500 ppm
Aug 07 02:23:36 dist-test-slave-69rg krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:23:36.378254 16786 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:23:36.379482 16786 webserver.cc:460] Webserver started at http://127.4.221.66:44961/ using document root <none> and password file <none>
I20260807 02:23:36.380256 16786 fs_manager.cc:359] Metadata directory not provided
I20260807 02:23:36.380394 16786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:23:36.380698 16786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:23:36 dist-test-slave-69rg krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:23:36.397528 16786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "de4dec8e94464c86923e9e64996e9515"
format_stamp: "Formatted at 2026-08-07 02:23:36 on dist-test-slave-69rg"
server_key: "5d9008061c11c4758de469a53231d295"
server_key_iv: "250f6dcec86d96028376109626a57d84"
server_key_version: "kuduclusterkey@0"
I20260807 02:23:36.398509 16786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "de4dec8e94464c86923e9e64996e9515"
format_stamp: "Formatted at 2026-08-07 02:23:36 on dist-test-slave-69rg"
server_key: "5d9008061c11c4758de469a53231d295"
server_key_iv: "250f6dcec86d96028376109626a57d84"
server_key_version: "kuduclusterkey@0"
I20260807 02:23:36.404342 16786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:23:36.429440 16803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:23:36.431337 16786 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.002s
I20260807 02:23:36.431566 16786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "de4dec8e94464c86923e9e64996e9515"
format_stamp: "Formatted at 2026-08-07 02:23:36 on dist-test-slave-69rg"
server_key: "5d9008061c11c4758de469a53231d295"
server_key_iv: "250f6dcec86d96028376109626a57d84"
server_key_version: "kuduclusterkey@0"
I20260807 02:23:36.431828 16786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:23:36.457229 16786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:23:36.461716 16786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:23:36.462024 16786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:23:36.462729 16786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:23:36.463773 16786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:23:36.463892 16786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:23:36.463990 16786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:23:36.464056 16786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:23:36.474835 16786 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:45631
I20260807 02:23:36.476158 16786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:23:36.479002 16921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:45631 every 8 connection(s)
I20260807 02:23:36.479729 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 16786
I20260807 02:23:36.479867 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:23:36.480211 4981 external_mini_cluster.cc:1428] Setting key 77ba222c363bee5fa7ce438f181bf8bf
Aug 07 02:23:36 dist-test-slave-69rg krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:23:36.501502 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:36.480694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:44155 (local address 127.4.221.126:39495)
0807 02:23:36.480838 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 02:23:36.480843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:23:36.485648 (+ 4805us) server_negotiation.cc:407] Connection header received
0807 02:23:36.486894 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:23:36.486897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:23:36.486955 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:23:36.487011 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:23:36.488689 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.489461 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.491191 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.491684 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.491836 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:36.497326 (+ 5490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:23:36.497330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:23:36.497332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:23:36.497361 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:23:36.499100 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:23:36.499717 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:23:36.499721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:23:36.499723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:23:36.499813 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:23:36.500225 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:23:36.500228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:23:36.500230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:23:36.500458 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:23:36.500596 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 02:23:36.501146 (+ 550us) server_negotiation.cc:299] Negotiation successful
0807 02:23:36.501301 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 02:23:36.502526 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:36.481197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:39495 (local address 127.4.221.66:44155)
0807 02:23:36.485431 (+ 4234us) negotiation.cc:107] Waiting for socket to connect
0807 02:23:36.485496 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 02:23:36.486614 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:23:36.487447 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:23:36.487457 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:23:36.487843 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:23:36.488499 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:36.488517 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.489635 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.489638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:36.491030 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:36.491042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.492080 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.492086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:36.492358 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:36.493318 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:23:36.493354 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:23:36.495750 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:23:36.499254 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:23:36.499260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:23:36.499278 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:23:36.499579 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:23:36.499934 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:23:36.499937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:23:36.499939 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:23:36.500104 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:23:36.500592 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:23:36.500599 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:23:36.500884 (+ 285us) client_negotiation.cc:769] Sending connection context
0807 02:23:36.501081 (+ 197us) client_negotiation.cc:240] Negotiation successful
0807 02:23:36.501443 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3988,"thread_start_us":143,"threads_started":1}
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
I20260807 02:23:36.503768 16922 heartbeater.cc:347] Connected to a master server at 127.4.221.126:39495
I20260807 02:23:36.504052 16922 heartbeater.cc:464] Registering TS with master...
I20260807 02:23:36.504580 16922 heartbeater.cc:511] Master 127.4.221.126:39495 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
I20260807 02:23:36.505779 16558 ts_manager.cc:194] Registered new tserver with Master: de4dec8e94464c86923e9e64996e9515 (127.4.221.66:45631)
I20260807 02:23:36.506476 16558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:44155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:23:36.525367 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.221.84:38975/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:39495
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:23:36.721946 16927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:23:36.722383 16927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:23:36.722499 16927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:23:36.730008 16927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:23:36.730159 16927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:23:36.730335 16927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:23:36.746789 16927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.221.84:38975/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:39495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:23:36.748512 16927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:23:36.750133 16927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:23:36.767587 16933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:23:36.767587 16935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:23:36.769367 16927 server_base.cc:1034] running on GCE node
W20260807 02:23:36.770025 16932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:23:36.770793 16927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:23:36.771756 16927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Aug 07 02:23:36 dist-test-slave-69rg krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:23:36.779201 16927 hybrid_clock.cc:648] HybridClock initialized: now 1786069416779103 us; error 123 us; skew 500 ppm
Aug 07 02:23:36 dist-test-slave-69rg krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:23:36.784104 16927 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:23:36.785634 16927 webserver.cc:460] Webserver started at http://127.4.221.67:40189/ using document root <none> and password file <none>
I20260807 02:23:36.786734 16927 fs_manager.cc:359] Metadata directory not provided
I20260807 02:23:36.786918 16927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:23:36.787266 16927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:23:36 dist-test-slave-69rg krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:23:36.823184 16927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "c25fb859f44348fab961f8ae53322cd6"
format_stamp: "Formatted at 2026-08-07 02:23:36 on dist-test-slave-69rg"
server_key: "9b4d7d581ba0acc554fa8b78488703e4"
server_key_iv: "608169ee613a716a994aab31737b8f1e"
server_key_version: "kuduclusterkey@0"
I20260807 02:23:36.824504 16927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "c25fb859f44348fab961f8ae53322cd6"
format_stamp: "Formatted at 2026-08-07 02:23:36 on dist-test-slave-69rg"
server_key: "9b4d7d581ba0acc554fa8b78488703e4"
server_key_iv: "608169ee613a716a994aab31737b8f1e"
server_key_version: "kuduclusterkey@0"
I20260807 02:23:36.830929 16927 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260807 02:23:36.863464 16943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:23:36.865324 16927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:23:36.865484 16927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "c25fb859f44348fab961f8ae53322cd6"
format_stamp: "Formatted at 2026-08-07 02:23:36 on dist-test-slave-69rg"
server_key: "9b4d7d581ba0acc554fa8b78488703e4"
server_key_iv: "608169ee613a716a994aab31737b8f1e"
server_key_version: "kuduclusterkey@0"
I20260807 02:23:36.865639 16927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:23:36.895457 16927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:23:36.898880 16927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:23:36.899142 16927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:23:36.899837 16927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:23:36.900836 16927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:23:36.900910 16927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:23:36.901016 16927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:23:36.901062 16927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:23:36.913067 16927 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:44461
I20260807 02:23:36.913089 17059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:44461 every 8 connection(s)
I20260807 02:23:36.914352 16927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:23:36.924170 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 16927
I20260807 02:23:36.924324 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
Aug 07 02:23:36 dist-test-slave-69rg krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:23:36.924690 4981 external_mini_cluster.cc:1428] Setting key b1675772318a86ef7ed0a15262ad29ce
I20260807 02:23:36.929852 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:36.916368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:40239 (local address 127.4.221.126:39495)
0807 02:23:36.916556 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0807 02:23:36.916562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:23:36.917465 (+ 903us) server_negotiation.cc:407] Connection header received
0807 02:23:36.918593 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:23:36.918596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:23:36.918670 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:23:36.918761 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:23:36.919961 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.920511 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.921693 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.922048 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.922191 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:36.925707 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:23:36.925711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:23:36.925716 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:23:36.925758 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:23:36.927657 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:23:36.928214 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:23:36.928219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:23:36.928221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:23:36.928312 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:23:36.928687 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:23:36.928690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:23:36.928692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:23:36.928897 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:23:36.929007 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 02:23:36.929533 (+ 526us) server_negotiation.cc:299] Negotiation successful
0807 02:23:36.929657 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 02:23:36.931027 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:36.916803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:39495 (local address 127.4.221.67:40239)
0807 02:23:36.917307 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0807 02:23:36.917360 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 02:23:36.918358 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:23:36.919014 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:23:36.919021 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:23:36.919304 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:23:36.919805 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:36.919819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.920656 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.920661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:36.921543 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:36.921553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.922240 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.922244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:36.922457 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:36.923193 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:23:36.923210 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:23:36.925479 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:23:36.927781 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:23:36.927788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:23:36.927808 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:23:36.928099 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:23:36.928411 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:23:36.928415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:23:36.928417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:23:36.928572 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:23:36.929008 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:23:36.929016 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:23:36.929289 (+ 273us) client_negotiation.cc:769] Sending connection context
0807 02:23:36.929493 (+ 204us) client_negotiation.cc:240] Negotiation successful
0807 02:23:36.929883 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":151,"threads_started":1}
I20260807 02:23:36.935137 17060 heartbeater.cc:347] Connected to a master server at 127.4.221.126:39495
I20260807 02:23:36.935429 17060 heartbeater.cc:464] Registering TS with master...
I20260807 02:23:36.936034 17060 heartbeater.cc:511] Master 127.4.221.126:39495 requested a full tablet report, sending...
I20260807 02:23:36.937330 16558 ts_manager.cc:194] Registered new tserver with Master: c25fb859f44348fab961f8ae53322cd6 (127.4.221.67:44461)
I20260807 02:23:36.938086 16558 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:40239
I20260807 02:23:36.939364 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:23:36.949918 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:36.941774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.4.221.126:39495)
0807 02:23:36.941945 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 02:23:36.941951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:23:36.942123 (+ 172us) server_negotiation.cc:407] Connection header received
0807 02:23:36.942233 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:23:36.942236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:23:36.942283 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:23:36.942380 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:23:36.943169 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.943893 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.944712 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.945007 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.945127 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:36.946592 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:23:36.946596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:23:36.946598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:23:36.946627 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:23:36.948235 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:23:36.948653 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:23:36.948657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:23:36.948659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:23:36.948748 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:23:36.949026 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:23:36.949030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:23:36.949032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:23:36.949265 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:23:36.949399 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 02:23:36.949606 (+ 207us) server_negotiation.cc:299] Negotiation successful
0807 02:23:36.949730 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 02:23:36.953027 16558 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:23:36.956315 16558 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:23:36.975392 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:36.968329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:39671 (local address 127.0.0.1:39184)
0807 02:23:36.969481 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0807 02:23:36.969493 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 02:23:36.969563 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:23:36.970229 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:23:36.970236 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:23:36.970258 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:23:36.970382 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:36.970388 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.971816 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.971820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:36.973343 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:36.973358 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.974738 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.974742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:36.974855 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:36.974960 (+ 105us) client_negotiation.cc:769] Sending connection context
0807 02:23:36.975080 (+ 120us) client_negotiation.cc:240] Negotiation successful
0807 02:23:36.975228 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1094,"mutex_wait_us":29,"thread_start_us":93,"threads_started":1}
I20260807 02:23:36.976184 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:36.968622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.4.221.65:39671)
0807 02:23:36.969057 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0807 02:23:36.969064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:23:36.969670 (+ 606us) server_negotiation.cc:407] Connection header received
0807 02:23:36.969752 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:23:36.969757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:23:36.969953 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:23:36.970077 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:23:36.970504 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.971656 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.973498 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.974332 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.974435 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:36.974673 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0807 02:23:36.975850 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0807 02:23:36.975961 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":96,"threads_started":1}
I20260807 02:23:36.976399 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:36.966318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.4.221.66:45631)
0807 02:23:36.966774 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0807 02:23:36.966792 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 02:23:36.969039 (+ 2247us) server_negotiation.cc:407] Connection header received
0807 02:23:36.969125 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:23:36.969131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:23:36.969346 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:23:36.969498 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:23:36.970755 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.972007 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.974484 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.975306 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.975415 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:36.975551 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 02:23:36.976149 (+ 598us) server_negotiation.cc:299] Negotiation successful
0807 02:23:36.976255 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":110,"threads_started":1}
I20260807 02:23:36.976044 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:36.966200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:45631 (local address 127.0.0.1:40760)
0807 02:23:36.968700 (+ 2500us) negotiation.cc:107] Waiting for socket to connect
0807 02:23:36.968755 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 02:23:36.968897 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:23:36.970459 (+ 1562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:23:36.970462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:23:36.970482 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:23:36.970593 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:36.970600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.972148 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.972151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:36.974321 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:36.974332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.975633 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.975637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:36.975746 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:36.975770 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 02:23:36.975843 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 02:23:36.975917 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2309,"thread_start_us":129,"threads_started":1}
I20260807 02:23:36.980149 16711 tablet_service.cc:1467] Processing CreateTablet for tablet 79ec30b9057a4c15b1576da3fc2448ef (DEFAULT_TABLE table=test-table [id=eb324cb42c9449ef86b6201859f54127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:23:36.983444 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:36.967514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:44461 (local address 127.0.0.1:59198)
0807 02:23:36.969201 (+ 1687us) negotiation.cc:107] Waiting for socket to connect
0807 02:23:36.969215 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:23:36.969305 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:23:36.978740 (+ 9435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:23:36.978743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:23:36.978764 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:23:36.978857 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:36.978863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.980618 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.980622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:36.981865 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:36.981877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.983043 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.983047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:36.983141 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:36.983181 (+ 40us) client_negotiation.cc:769] Sending connection context
0807 02:23:36.983231 (+ 50us) client_negotiation.cc:240] Negotiation successful
0807 02:23:36.983301 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1618,"thread_start_us":109,"threads_started":1}
I20260807 02:23:36.983095 16853 tablet_service.cc:1467] Processing CreateTablet for tablet 79ec30b9057a4c15b1576da3fc2448ef (DEFAULT_TABLE table=test-table [id=eb324cb42c9449ef86b6201859f54127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:23:36.984267 16853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ec30b9057a4c15b1576da3fc2448ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:23:36.984370 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:36.967954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.4.221.67:44461)
0807 02:23:36.977109 (+ 9155us) server_negotiation.cc:206] Beginning negotiation
0807 02:23:36.977123 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:23:36.977149 (+ 26us) server_negotiation.cc:407] Connection header received
0807 02:23:36.977237 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:23:36.977243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:23:36.977453 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:23:36.979011 (+ 1558us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:23:36.979179 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.980480 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.982080 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:36.982901 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:36.983925 (+ 1024us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:36.984072 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 02:23:36.984122 (+ 50us) server_negotiation.cc:299] Negotiation successful
0807 02:23:36.984226 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8993,"thread_start_us":149,"threads_started":1}
I20260807 02:23:36.985994 16711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ec30b9057a4c15b1576da3fc2448ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:23:36.987377 16974 tablet_service.cc:1467] Processing CreateTablet for tablet 79ec30b9057a4c15b1576da3fc2448ef (DEFAULT_TABLE table=test-table [id=eb324cb42c9449ef86b6201859f54127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:23:36.988375 16974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79ec30b9057a4c15b1576da3fc2448ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:23:36.997839 17078 tablet_bootstrap.cc:492] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6: Bootstrap starting.
I20260807 02:23:37.001091 17078 tablet_bootstrap.cc:654] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6: Neither blocks nor log segments found. Creating new log.
I20260807 02:23:37.002408 17078 log.cc:824] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6: Log is configured to *not* fsync() on all Append() calls
I20260807 02:23:37.005175 17078 tablet_bootstrap.cc:492] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6: No bootstrap required, opened a new log
I20260807 02:23:37.005386 17078 ts_tablet_manager.cc:1397] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 02:23:37.007025 17079 tablet_bootstrap.cc:492] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2: Bootstrap starting.
I20260807 02:23:37.009433 17079 tablet_bootstrap.cc:654] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2: Neither blocks nor log segments found. Creating new log.
I20260807 02:23:37.009342 17078 raft_consensus.cc:348] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4dec8e94464c86923e9e64996e9515" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 45631 } } peers { permanent_uuid: "c25fb859f44348fab961f8ae53322cd6" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44461 } } peers { permanent_uuid: "2b0982a754cd4baab28266c18f1315e2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39671 } }
I20260807 02:23:37.009604 17078 raft_consensus.cc:374] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:23:37.009667 17078 raft_consensus.cc:729] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25fb859f44348fab961f8ae53322cd6, State: Initialized, Role: FOLLOWER
I20260807 02:23:37.010428 17078 consensus_queue.cc:260] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4dec8e94464c86923e9e64996e9515" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 45631 } } peers { permanent_uuid: "c25fb859f44348fab961f8ae53322cd6" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44461 } } peers { permanent_uuid: "2b0982a754cd4baab28266c18f1315e2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39671 } }
I20260807 02:23:37.011602 17078 ts_tablet_manager.cc:1428] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 02:23:37.011754 17081 tablet_bootstrap.cc:492] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515: Bootstrap starting.
I20260807 02:23:37.012013 17060 heartbeater.cc:503] Master 127.4.221.126:39495 was elected leader, sending a full tablet report...
I20260807 02:23:37.014196 17081 tablet_bootstrap.cc:654] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515: Neither blocks nor log segments found. Creating new log.
I20260807 02:23:37.015740 17081 log.cc:824] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515: Log is configured to *not* fsync() on all Append() calls
I20260807 02:23:37.015978 17079 log.cc:824] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2: Log is configured to *not* fsync() on all Append() calls
I20260807 02:23:37.018778 17081 tablet_bootstrap.cc:492] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515: No bootstrap required, opened a new log
I20260807 02:23:37.018806 17079 tablet_bootstrap.cc:492] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2: No bootstrap required, opened a new log
I20260807 02:23:37.019022 17081 ts_tablet_manager.cc:1397] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260807 02:23:37.019045 17079 ts_tablet_manager.cc:1397] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260807 02:23:37.023011 17081 raft_consensus.cc:348] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4dec8e94464c86923e9e64996e9515" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 45631 } } peers { permanent_uuid: "c25fb859f44348fab961f8ae53322cd6" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44461 } } peers { permanent_uuid: "2b0982a754cd4baab28266c18f1315e2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39671 } }
I20260807 02:23:37.023368 17081 raft_consensus.cc:374] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:23:37.023459 17081 raft_consensus.cc:729] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de4dec8e94464c86923e9e64996e9515, State: Initialized, Role: FOLLOWER
I20260807 02:23:37.024009 17081 consensus_queue.cc:260] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4dec8e94464c86923e9e64996e9515" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 45631 } } peers { permanent_uuid: "c25fb859f44348fab961f8ae53322cd6" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44461 } } peers { permanent_uuid: "2b0982a754cd4baab28266c18f1315e2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39671 } }
I20260807 02:23:37.024741 17079 raft_consensus.cc:348] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4dec8e94464c86923e9e64996e9515" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 45631 } } peers { permanent_uuid: "c25fb859f44348fab961f8ae53322cd6" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44461 } } peers { permanent_uuid: "2b0982a754cd4baab28266c18f1315e2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39671 } }
I20260807 02:23:37.025067 17079 raft_consensus.cc:374] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:23:37.025116 17081 ts_tablet_manager.cc:1428] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 02:23:37.025170 17079 raft_consensus.cc:729] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b0982a754cd4baab28266c18f1315e2, State: Initialized, Role: FOLLOWER
I20260807 02:23:37.025614 16922 heartbeater.cc:503] Master 127.4.221.126:39495 was elected leader, sending a full tablet report...
I20260807 02:23:37.025660 17079 consensus_queue.cc:260] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4dec8e94464c86923e9e64996e9515" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 45631 } } peers { permanent_uuid: "c25fb859f44348fab961f8ae53322cd6" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44461 } } peers { permanent_uuid: "2b0982a754cd4baab28266c18f1315e2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39671 } }
I20260807 02:23:37.026693 17079 ts_tablet_manager.cc:1428] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260807 02:23:37.027036 16780 heartbeater.cc:503] Master 127.4.221.126:39495 was elected leader, sending a full tablet report...
I20260807 02:23:37.059975 17086 raft_consensus.cc:482] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:23:37.060359 17086 raft_consensus.cc:504] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4dec8e94464c86923e9e64996e9515" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 45631 } } peers { permanent_uuid: "c25fb859f44348fab961f8ae53322cd6" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44461 } } peers { permanent_uuid: "2b0982a754cd4baab28266c18f1315e2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39671 } }
I20260807 02:23:37.062614 17086 leader_election.cc:288] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de4dec8e94464c86923e9e64996e9515 (127.4.221.66:45631), c25fb859f44348fab961f8ae53322cd6 (127.4.221.67:44461)
I20260807 02:23:37.068058 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:37.062743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:48819 (local address 127.4.221.66:45631)
0807 02:23:37.062866 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0807 02:23:37.062870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:23:37.062979 (+ 109us) server_negotiation.cc:407] Connection header received
0807 02:23:37.063173 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:23:37.063177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:23:37.063233 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:23:37.063282 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:23:37.063738 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:37.064515 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:37.066537 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:37.067226 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:37.067337 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:37.067394 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 02:23:37.067804 (+ 410us) server_negotiation.cc:299] Negotiation successful
0807 02:23:37.067895 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:23:37.068146 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:37.062322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:37365 (local address 127.4.221.67:44461)
0807 02:23:37.062631 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0807 02:23:37.062637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:23:37.063274 (+ 637us) server_negotiation.cc:407] Connection header received
0807 02:23:37.063471 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:23:37.063474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:23:37.063525 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:23:37.063573 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:23:37.064205 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:37.065003 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:37.066715 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:37.067366 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:37.067468 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:37.067521 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0807 02:23:37.067907 (+ 386us) server_negotiation.cc:299] Negotiation successful
0807 02:23:37.068009 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260807 02:23:37.068602 17087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:37.062127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:44461 (local address 127.4.221.65:37365)
0807 02:23:37.063172 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0807 02:23:37.063186 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:23:37.063350 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:23:37.063878 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:23:37.063881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:23:37.063902 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:23:37.064038 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:37.064046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:37.065198 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:37.065202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:37.066541 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:37.066553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:37.067624 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:37.067640 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:37.067753 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:37.067776 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 02:23:37.068354 (+ 578us) client_negotiation.cc:240] Negotiation successful
0807 02:23:37.068436 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":245,"thread_start_us":102,"threads_started":1}
I20260807 02:23:37.069396 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:37.062126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:45631 (local address 127.4.221.65:48819)
0807 02:23:37.062884 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0807 02:23:37.062901 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:23:37.063029 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:23:37.063434 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:23:37.063437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:23:37.063454 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:23:37.063595 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:37.063601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:37.064789 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:37.064793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:37.066360 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:23:37.066374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:23:37.067517 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:23:37.067521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:23:37.067644 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:37.067669 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 02:23:37.068659 (+ 990us) client_negotiation.cc:240] Negotiation successful
0807 02:23:37.068747 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":142,"threads_started":1}
I20260807 02:23:37.069620 16995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ec30b9057a4c15b1576da3fc2448ef" candidate_uuid: "2b0982a754cd4baab28266c18f1315e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25fb859f44348fab961f8ae53322cd6" is_pre_election: true
I20260807 02:23:37.070039 16995 raft_consensus.cc:2454] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0982a754cd4baab28266c18f1315e2 in term 0.
I20260807 02:23:37.070200 16874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ec30b9057a4c15b1576da3fc2448ef" candidate_uuid: "2b0982a754cd4baab28266c18f1315e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de4dec8e94464c86923e9e64996e9515" is_pre_election: true
I20260807 02:23:37.070492 16874 raft_consensus.cc:2454] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b0982a754cd4baab28266c18f1315e2 in term 0.
I20260807 02:23:37.070634 16660 leader_election.cc:302] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0982a754cd4baab28266c18f1315e2, c25fb859f44348fab961f8ae53322cd6; no voters:
I20260807 02:23:37.071288 17086 raft_consensus.cc:2781] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:23:37.071403 17086 raft_consensus.cc:482] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:23:37.071467 17086 raft_consensus.cc:3037] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:23:37.072947 17086 raft_consensus.cc:504] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4dec8e94464c86923e9e64996e9515" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 45631 } } peers { permanent_uuid: "c25fb859f44348fab961f8ae53322cd6" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44461 } } peers { permanent_uuid: "2b0982a754cd4baab28266c18f1315e2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39671 } }
I20260807 02:23:37.073745 16874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ec30b9057a4c15b1576da3fc2448ef" candidate_uuid: "2b0982a754cd4baab28266c18f1315e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de4dec8e94464c86923e9e64996e9515"
I20260807 02:23:37.073933 16874 raft_consensus.cc:3037] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:23:37.075451 16874 raft_consensus.cc:2454] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0982a754cd4baab28266c18f1315e2 in term 1.
I20260807 02:23:37.075819 17086 leader_election.cc:288] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [CANDIDATE]: Term 1 election: Requested vote from peers de4dec8e94464c86923e9e64996e9515 (127.4.221.66:45631), c25fb859f44348fab961f8ae53322cd6 (127.4.221.67:44461)
I20260807 02:23:37.075882 16662 leader_election.cc:302] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b0982a754cd4baab28266c18f1315e2, de4dec8e94464c86923e9e64996e9515; no voters:
I20260807 02:23:37.076104 17086 raft_consensus.cc:2781] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:23:37.076452 17086 raft_consensus.cc:686] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [term 1 LEADER]: Becoming Leader. State: Replica: 2b0982a754cd4baab28266c18f1315e2, State: Running, Role: LEADER
I20260807 02:23:37.076639 16995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79ec30b9057a4c15b1576da3fc2448ef" candidate_uuid: "2b0982a754cd4baab28266c18f1315e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25fb859f44348fab961f8ae53322cd6"
I20260807 02:23:37.076785 16995 raft_consensus.cc:3037] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:23:37.077054 17086 consensus_queue.cc:237] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4dec8e94464c86923e9e64996e9515" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 45631 } } peers { permanent_uuid: "c25fb859f44348fab961f8ae53322cd6" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44461 } } peers { permanent_uuid: "2b0982a754cd4baab28266c18f1315e2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39671 } }
I20260807 02:23:37.078329 16995 raft_consensus.cc:2454] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b0982a754cd4baab28266c18f1315e2 in term 1.
I20260807 02:23:37.082077 16558 catalog_manager.cc:5495] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b0982a754cd4baab28266c18f1315e2 (127.4.221.65). New cstate: current_term: 1 leader_uuid: "2b0982a754cd4baab28266c18f1315e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de4dec8e94464c86923e9e64996e9515" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 45631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c25fb859f44348fab961f8ae53322cd6" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b0982a754cd4baab28266c18f1315e2" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39671 } health_report { overall_health: HEALTHY } } }
I20260807 02:23:37.104440 17077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:37.099916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.4.221.65:39671)
0807 02:23:37.100038 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0807 02:23:37.100042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:23:37.100057 (+ 15us) server_negotiation.cc:407] Connection header received
0807 02:23:37.100150 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:23:37.100155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:23:37.100248 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:23:37.100369 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:23:37.100788 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:37.101389 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:37.102384 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:23:37.102627 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:23:37.102743 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:23:37.103066 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:23:37.103636 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:23:37.103766 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 02:23:37.104226 (+ 460us) server_negotiation.cc:299] Negotiation successful
0807 02:23:37.104293 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260807 02:23:37.112440 16995 raft_consensus.cc:1261] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6 [term 1 FOLLOWER]: Refusing update from remote peer 2b0982a754cd4baab28266c18f1315e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:23:37.113411 16874 raft_consensus.cc:1261] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515 [term 1 FOLLOWER]: Refusing update from remote peer 2b0982a754cd4baab28266c18f1315e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:23:37.114271 17086 consensus_queue.cc:1035] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de4dec8e94464c86923e9e64996e9515" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 45631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:23:37.114790 17086 consensus_queue.cc:1035] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c25fb859f44348fab961f8ae53322cd6" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 44461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260807 02:23:37.118678 16781 tablet.cc:2369] T 79ec30b9057a4c15b1576da3fc2448ef P 2b0982a754cd4baab28266c18f1315e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:23:37.121984 17092 mvcc.cc:205] Tried to move back new op lower bound from 7315740332479381504 to 7315740332352978944. Current Snapshot: MvccSnapshot[applied={T|T < 7315740332479381504}]
I20260807 02:23:37.132642 17095 mvcc.cc:205] Tried to move back new op lower bound from 7315740332479381504 to 7315740332352978944. Current Snapshot: MvccSnapshot[applied={T|T < 7315740332479381504}]
I20260807 02:23:37.139047 16557 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:23:37.139245 16557 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:23:37.139912 17093 mvcc.cc:205] Tried to move back new op lower bound from 7315740332479381504 to 7315740332352978944. Current Snapshot: MvccSnapshot[applied={T|T < 7315740332479381504}]
I20260807 02:23:37.142922 16557 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74bc7c1d9cac444195fa12112de5e170: Sending DeleteTablet for 3 replicas of tablet 79ec30b9057a4c15b1576da3fc2448ef
I20260807 02:23:37.144399 16853 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ec30b9057a4c15b1576da3fc2448ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:23:37 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:40760
I20260807 02:23:37.144922 16711 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ec30b9057a4c15b1576da3fc2448ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:23:37 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:39184
I20260807 02:23:37.145373 16974 tablet_service.cc:1514] Processing DeleteTablet for tablet 79ec30b9057a4c15b1576da3fc2448ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:23:37 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:59198
I20260807 02:23:37.147516 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 16640
I20260807 02:23:37.150503 17106 tablet_replica.cc:330] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6: stopping tablet replica
I20260807 02:23:37.150846 17104 tablet_replica.cc:330] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515: stopping tablet replica
I20260807 02:23:37.151379 17106 raft_consensus.cc:2229] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:23:37.151826 17104 raft_consensus.cc:2229] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:23:37.152217 17104 raft_consensus.cc:2258] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:23:37.152716 17106 raft_consensus.cc:2258] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:23:37.154588 17106 ts_tablet_manager.cc:1905] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:23:37.160378 17104 ts_tablet_manager.cc:1905] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:23:37.163626 17104 ts_tablet_manager.cc:1918] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:23:37.163868 17104 log.cc:1196] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/79ec30b9057a4c15b1576da3fc2448ef
I20260807 02:23:37.164348 17104 ts_tablet_manager.cc:1939] T 79ec30b9057a4c15b1576da3fc2448ef P de4dec8e94464c86923e9e64996e9515: Deleting consensus metadata
W20260807 02:23:37.166330 16545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:39671 (error 108)
I20260807 02:23:37.166957 17106 ts_tablet_manager.cc:1918] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:23:37.166939 16545 catalog_manager.cc:4841] TS de4dec8e94464c86923e9e64996e9515 (127.4.221.66:45631): tablet 79ec30b9057a4c15b1576da3fc2448ef (table test-table [id=eb324cb42c9449ef86b6201859f54127]) successfully deleted
I20260807 02:23:37.167498 17106 log.cc:1196] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/79ec30b9057a4c15b1576da3fc2448ef
I20260807 02:23:37.167696 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 16786
I20260807 02:23:37.168818 17106 ts_tablet_manager.cc:1939] T 79ec30b9057a4c15b1576da3fc2448ef P c25fb859f44348fab961f8ae53322cd6: Deleting consensus metadata
I20260807 02:23:37.170504 16543 catalog_manager.cc:4841] TS c25fb859f44348fab961f8ae53322cd6 (127.4.221.67:44461): tablet 79ec30b9057a4c15b1576da3fc2448ef (table test-table [id=eb324cb42c9449ef86b6201859f54127]) successfully deleted
I20260807 02:23:37.172449 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:23:37.172045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:39671 (local address 127.0.0.1:39198)
0807 02:23:37.172257 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0807 02:23:37.172338 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260807 02:23:37.172674 16545 catalog_manager.cc:4584] TS 2b0982a754cd4baab28266c18f1315e2 (127.4.221.65:39671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79ec30b9057a4c15b1576da3fc2448ef: Network error: Client connection negotiation failed: client connection to 127.4.221.65:39671: connect: Connection refused (error 111)
Aug 07 02:23:37 dist-test-slave-69rg krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069416, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:23:37.178278 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 16927
I20260807 02:23:37.187863 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 16525
I20260807 02:23:37.197001 4981 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 02:23:37.542541 4981 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 02:23:37.542681 4981 mini_ranger.cc:66] Stopping Ranger...
I20260807 02:23:37.888584 4981 mini_ranger.cc:68] Stopped Ranger
2026-08-07 02:23:37.888 UTC [16096] LOG: received smart shutdown request
2026-08-07 02:23:37.893 UTC [16096] LOG: background worker "logical replication launcher" (PID 16105) exited with exit code 1
2026-08-07 02:23:37.893 UTC [16100] LOG: shutting down
2026-08-07 02:23:38.097 UTC [16096] LOG: database system is shut down
2026-08-07T02:23:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50475 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:23:38 dist-test-slave-69rg krb5kdc[17117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:23:38 dist-test-slave-69rg krb5kdc[17117](info): set up 2 sockets
Aug 07 02:23:38 dist-test-slave-69rg krb5kdc[17117](info): commencing operation
krb5kdc: starting...
W20260807 02:23:39.365899 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.003s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:23:39 dist-test-slave-69rg krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:23:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:23:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/rangeradmin_127.4.221.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/rangeradmin_127.4.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/rangerlookup_127.4.221.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/rangerlookup_127.4.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/HTTP_127.4.221.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/HTTP_127.4.221.84@KRBTEST.COM.keytab.
I20260807 02:23:39.557197 4981 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskBkVddJ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/postgres -l logfile start
2026-08-07 02:23:42.310 UTC [17147] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-07 02:23:42.311 UTC [17147] LOG: listening on IPv4 address "127.4.221.84", port 37847
2026-08-07 02:23:42.315 UTC [17147] LOG: listening on Unix socket "/tmp/.s.PGSQL.37847"
2026-08-07 02:23:42.337 UTC [17150] LOG: database system was shut down at 2026-08-07 02:23:40 UTC
2026-08-07 02:23:42.341 UTC [17147] LOG: database system is ready to accept connections
I20260807 02:23:43.310355 4981 mini_postgres.cc:96] Postgres bound to 37847
2026-08-07 02:23:43.317 UTC [17158] FATAL: database "slave" does not exist
127.4.221.84:37847 - accepting connections
I20260807 02:23:43.318483 4981 mini_ranger.cc:161] Starting Ranger...
I20260807 02:23:43.335026 4981 mini_ranger.cc:84] Created miniranger Postgres user
I20260807 02:23:43.846244 4981 mini_ranger.cc:87] Created ranger Postgres database
I20260807 02:23:43.846338 4981 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-admin
2026-08-07 02:23:44,117 [I] DB FLAVOR :POSTGRES
2026-08-07 02:23:44,117 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 02:23:44,118 [I] Checking connection..
2026-08-07 02:23:44,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-07 02:23:44,369 [I] Checking connection passed.
2026-08-07 02:23:44,370 [I] --------- Verifying version history table ---------
2026-08-07 02:23:44,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 02:23:44,637 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 02:23:44,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 02:23:44,881 [I] Table x_db_version_h does not exist in database ranger
2026-08-07 02:23:44,881 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-07 02:23:44,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-07 02:23:45.134 UTC [17223] WARNING: there is no transaction in progress
2026-08-07 02:23:45,141 [I] create_dbversion_catalog.sql file imported successfully
2026-08-07 02:23:45,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-07 02:23:45,390 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-07 02:23:45,390 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 02:23:45,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-07 02:23:45,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-07 02:23:45,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-69rg.c.gcp-upstream.internal','N') ;"
2026-08-07 02:23:46,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 02:23:46,396 [I] Table x_portal_user does not exist in database ranger
2026-08-07 02:23:46,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 02:23:46,647 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-07 02:23:46,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 02:23:46,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 02:23:47,174 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-07 02:23:47,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-07 02:23:48.190 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:48.207 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:48.224 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:48.241 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:48.262 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:48.281 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:48.337 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:48.344 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:48.359 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:48.369 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:48.380 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:48.394 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:48.404 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:48.411 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:48.418 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:48.425 UTC [17335] WARNING: there is no transaction in progress
2026-08-07 02:23:49,110 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-07 02:23:49,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-07 02:23:49,342 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-07 02:23:49,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-07 02:23:49,578 [I] Table x_portal_user already exists in database 'ranger'
2026-08-07 02:23:49,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-07 02:23:49,811 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-07 02:23:49,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-07 02:23:50,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-07 02:23:50,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-69rg.c.gcp-upstream.internal';"
2026-08-07 02:23:50,571 [I] CORE_DB_SCHEMA import status has been updated
2026-08-07 02:23:50,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-07 02:23:50,826 [I] DB_PATCHES have already been applied
I20260807 02:23:50.832300 4981 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-admin:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskBkVddJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260807 02:23:50.832372 4981 mini_ranger.cc:193] Using host: 127.4.221.84
I20260807 02:23:50.836820 4981 mini_ranger.cc:227] Ranger admin URL: http://127.4.221.84:36471
Aug 07, 2026 2:23:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 2:23:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 2:23:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-admin/ews/webapp
Aug 07, 2026 2:23:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 07, 2026 2:23:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 07, 2026 2:23:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-admin/ews/webapp].
Aug 07, 2026 2:23:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260807 02:23:51.985078 4981 mini_ranger.cc:160] Time spent starting Ranger: real 8.667s user 0.000s sys 0.013s
Aug 07, 2026 2:24:13 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 07, 2026 2:24:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 07, 2026 2:24:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 07, 2026 2:24:13 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 07, 2026 2:24:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 07, 2026 2:24:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 07, 2026 2:24:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 07, 2026 2:24:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 07, 2026 2:24:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 07, 2026 2:24:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 07, 2026 2:24:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 07, 2026 2:24:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 07, 2026 2:24:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 07, 2026 2:24:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 07, 2026 2:24:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 07, 2026 2:24:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 07, 2026 2:24:13 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 07, 2026 2:24:14 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260807 02:24:14.772209 4981 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.221.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.221.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/rangerkms_127.4.221.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.221.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/rangerkms_127.4.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.221.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.221.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.221.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/HTTP_127.4.221.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.221.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/HTTP_127.4.221.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 07 02:24:14 dist-test-slave-69rg krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069454, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260807 02:24:14.868671 4981 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260807 02:24:14.880107 4981 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260807 02:24:15.894227 4981 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260807 02:24:15.894342 4981 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-kms
I20260807 02:24:15.894364 4981 mini_ranger_kms.cc:227] Using postgres at 127.4.221.84:37847
2026-08-07 02:24:16,143 [I] DB FLAVOR :POSTGRES
2026-08-07 02:24:16,143 [I] --------- Verifying Ranger DB connection ---------
2026-08-07 02:24:16,143 [I] Checking connection
2026-08-07 02:24:16,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-07 02:24:16,446 [I] connection success
2026-08-07 02:24:16,446 [I] --------- Verifying Ranger DB tables ---------
2026-08-07 02:24:16,446 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-07 02:24:16,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-07 02:24:16,725 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-07 02:24:16,725 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-07 02:24:16,725 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-07 02:24:16,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.221.84:37847/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-07 02:24:17,003 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260807 02:24:17.479903 4981 mini_ranger_kms.cc:310] Created kms service
I20260807 02:24:17.710433 4981 mini_ranger_kms.cc:326] Created kudu user
I20260807 02:24:17.761518 4981 mini_ranger_kms.cc:343] Created rangerkms user
Aug 07 02:24:17 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069454, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:24:18.228626 4981 mini_ranger_kms.cc:384] Added ranger policy
I20260807 02:24:18.228806 4981 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-kms:/tmp/dist-test-taskBkVddJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskBkVddJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260807 02:24:18.228837 4981 mini_ranger_kms.cc:240] Using host: 127.4.221.84
I20260807 02:24:18.234481 4981 mini_ranger_kms.cc:276] Ranger KMS PID: 17521
I20260807 02:24:18.234573 4981 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.221.84:53741
Aug 07, 2026 2:24:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 07, 2026 2:24:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 07, 2026 2:24:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 07, 2026 2:24:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 07, 2026 2:24:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-kms/ews/webapp].
Aug 07, 2026 2:24:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 07 02:24:20 dist-test-slave-69rg krb5kdc[17117](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786069460, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.221.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:20.594674 4981 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.726s user 0.012s sys 0.216s
I20260807 02:24:20.594828 4981 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260807 02:24:20.594918 4981 mini_ranger_kms.cc:401] 127.4.221.84:53741/kms/v1/keys
Aug 07 02:24:20 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069460, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.221.84@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
Aug 07 02:24:21 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069460, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.221.84@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
Aug 07 02:24:23 dist-test-slave-69rg krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:24:23.641705 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:35055
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:35055
--ranger_config_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.221.84:53741/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:23.755577 17574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:23.755831 17574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:23.755869 17574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:23.759999 17574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:24:23.760061 17574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:23.760079 17574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:23.760097 17574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:24:23.760111 17574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:24:23.765579 17574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.221.84:53741/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:35055
--ranger_config_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:35055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:23.766657 17574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:23.767692 17574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:23.777614 17580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:23.777741 17581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:23.777892 17574 server_base.cc:1034] running on GCE node
W20260807 02:24:23.777606 17583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:23.778496 17574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:23.779556 17574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:23.780761 17574 hybrid_clock.cc:648] HybridClock initialized: now 1786069463780743 us; error 38 us; skew 500 ppm
Aug 07 02:24:23 dist-test-slave-69rg krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069463, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:23.783903 17574 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:24:23.785004 17574 webserver.cc:460] Webserver started at http://127.4.221.126:44209/ using document root <none> and password file <none>
I20260807 02:24:23.785665 17574 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:23.785761 17574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:23.786037 17574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:24:23 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069463, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:24:23.844828 17574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "7f5483b891c645aebabee96557c6d391"
format_stamp: "Formatted at 2026-08-07 02:24:23 on dist-test-slave-69rg"
server_key: "d7fb51cd33d5a6330ca39fb2b8fa4a61"
server_key_iv: "1c1025525af7044b6fb3b18e3e24635f"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:23.845484 17574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "7f5483b891c645aebabee96557c6d391"
format_stamp: "Formatted at 2026-08-07 02:24:23 on dist-test-slave-69rg"
server_key: "d7fb51cd33d5a6330ca39fb2b8fa4a61"
server_key_iv: "1c1025525af7044b6fb3b18e3e24635f"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:23.849484 17574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260807 02:24:23.867262 17590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:23.868659 17574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:24:23.868810 17574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "7f5483b891c645aebabee96557c6d391"
format_stamp: "Formatted at 2026-08-07 02:24:23 on dist-test-slave-69rg"
server_key: "d7fb51cd33d5a6330ca39fb2b8fa4a61"
server_key_iv: "1c1025525af7044b6fb3b18e3e24635f"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:23.868952 17574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:23.892381 17574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:23.895303 17574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:23.895503 17574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:23.903555 17574 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:35055
I20260807 02:24:23.903597 17642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:35055 every 8 connection(s)
I20260807 02:24:23.904523 17574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:24:23.907882 17643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:23.910205 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 17574
I20260807 02:24:23.910365 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:24:23.910694 4981 external_mini_cluster.cc:1428] Setting key fdd17be719ff8c192689b59892d0604b
I20260807 02:24:23.915282 17643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391: Bootstrap starting.
I20260807 02:24:23.918015 17643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:23.918951 17643 log.cc:824] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391: Log is configured to *not* fsync() on all Append() calls
Aug 07 02:24:23 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:23.921231 17643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391: No bootstrap required, opened a new log
I20260807 02:24:23.925252 17647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:23.912116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.4.221.126:35055)
0807 02:24:23.912546 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:23.912564 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:23.912607 (+ 43us) server_negotiation.cc:407] Connection header received
0807 02:24:23.913241 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:23.913259 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:23.913562 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:23.913922 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:23.914759 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:23.915824 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:23.916694 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:23.917156 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:23.917325 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:23.920176 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:23.920182 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:23.920194 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:23.920219 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:23.922487 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:23.922989 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:23.922998 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:23.923007 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:23.923111 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:23.923418 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:23.923421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:23.923423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:23.923801 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:23.923999 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:23.924272 (+ 273us) server_negotiation.cc:299] Negotiation successful
0807 02:24:23.924557 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":158,"threads_started":1}
I20260807 02:24:23.925069 17643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5483b891c645aebabee96557c6d391" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35055 } }
I20260807 02:24:23.925400 17643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:23.925498 17643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5483b891c645aebabee96557c6d391, State: Initialized, Role: FOLLOWER
I20260807 02:24:23.926003 17643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5483b891c645aebabee96557c6d391" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35055 } }
I20260807 02:24:23.926155 17643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:24:23.926240 17643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:24:23.926338 17643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:23.927405 17643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5483b891c645aebabee96557c6d391" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35055 } }
I20260807 02:24:23.927825 17643 leader_election.cc:302] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f5483b891c645aebabee96557c6d391; no voters:
I20260807 02:24:23.928143 17643 leader_election.cc:288] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:24:23.928287 17649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:24:23.928571 17649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 1 LEADER]: Becoming Leader. State: Replica: 7f5483b891c645aebabee96557c6d391, State: Running, Role: LEADER
I20260807 02:24:23.929147 17649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5483b891c645aebabee96557c6d391" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35055 } }
I20260807 02:24:23.929302 17643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:24:23.930750 17650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f5483b891c645aebabee96557c6d391" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5483b891c645aebabee96557c6d391" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35055 } } }
I20260807 02:24:23.930869 17650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:23.931090 17650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f5483b891c645aebabee96557c6d391. Latest consensus state: current_term: 1 leader_uuid: "7f5483b891c645aebabee96557c6d391" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5483b891c645aebabee96557c6d391" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35055 } } }
I20260807 02:24:23.931165 17650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:23.931459 17643 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 02:24:23.934842 17663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:24:23.938550 17663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:24:23.944792 17663 catalog_manager.cc:1347] Generated new cluster ID: 350d2cb30d2b48da91edc808e03b5235
I20260807 02:24:23.944852 17663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:24:23.977596 17663 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:24:23.979075 17663 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:24:23.985849 17663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391: Generated new TSK 0
I20260807 02:24:23.986752 17663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:24:24.721498 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.221.84:53741/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:35055
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:24.871604 17690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:24.871968 17690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:24.872051 17690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:24.877384 17690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:24.877480 17690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:24.877588 17690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:24:24.883327 17690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.221.84:53741/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:35055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:24.884620 17690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:24.886008 17690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:24.898602 17698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:24.899060 17700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:24.899338 17697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:24.900477 17690 server_base.cc:1034] running on GCE node
I20260807 02:24:24.900916 17690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:24.901662 17690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:24.902840 17690 hybrid_clock.cc:648] HybridClock initialized: now 1786069464902808 us; error 45 us; skew 500 ppm
Aug 07 02:24:24 dist-test-slave-69rg krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069464, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:24.906517 17690 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:24:24.907513 17690 webserver.cc:460] Webserver started at http://127.4.221.65:46747/ using document root <none> and password file <none>
I20260807 02:24:24.908187 17690 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:24.908262 17690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:24.908470 17690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:24:24 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069464, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:24:24.927078 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "eb2fef31a22343e3a41fb1d6079b4336"
format_stamp: "Formatted at 2026-08-07 02:24:24 on dist-test-slave-69rg"
server_key: "f2250b8f986fcf98937eeb379878b326"
server_key_iv: "b0599d96f064c3eca168f37184d735c3"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:24.927922 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "eb2fef31a22343e3a41fb1d6079b4336"
format_stamp: "Formatted at 2026-08-07 02:24:24 on dist-test-slave-69rg"
server_key: "f2250b8f986fcf98937eeb379878b326"
server_key_iv: "b0599d96f064c3eca168f37184d735c3"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:24.936179 17690 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260807 02:24:24.952442 17709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:24.954145 17690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:24:24.954270 17690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "eb2fef31a22343e3a41fb1d6079b4336"
format_stamp: "Formatted at 2026-08-07 02:24:24 on dist-test-slave-69rg"
server_key: "f2250b8f986fcf98937eeb379878b326"
server_key_iv: "b0599d96f064c3eca168f37184d735c3"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:24.954389 17690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:24.978772 17690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:24.982509 17690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:24.982777 17690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:24.983485 17690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:24.984556 17690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:24:24.984673 17690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:24.984798 17690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:24:24.984881 17690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:24.998154 17690 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:45017
I20260807 02:24:24.999281 17690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:24:25.000201 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 17690
I20260807 02:24:25.000327 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:24:25.000682 4981 external_mini_cluster.cc:1428] Setting key d80f21a5b245e5b2b954c11db252990c
I20260807 02:24:25.000545 17824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:45017 every 8 connection(s)
Aug 07 02:24:25 dist-test-slave-69rg krb5kdc[17117](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786069465, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
Aug 07 02:24:25 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069464, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:25.025031 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:25.006931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:35055 (local address 127.4.221.65:34485)
0807 02:24:25.007416 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:25.007469 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:25.009250 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:25.012763 (+ 3513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:25.012773 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:25.013131 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:25.013916 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:25.013939 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:25.015096 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:25.015100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:25.016173 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:25.016185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:25.016890 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:25.016894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:25.017094 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:25.017899 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:25.017925 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:25.019782 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:25.021690 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:25.021697 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:25.021711 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:25.022094 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:25.022444 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:25.022447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:25.022449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:25.022617 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:25.023099 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:25.023106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:25.023415 (+ 309us) client_negotiation.cc:769] Sending connection context
0807 02:24:25.023609 (+ 194us) client_negotiation.cc:240] Negotiation successful
0807 02:24:25.023987 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":149,"threads_started":1}
I20260807 02:24:25.025853 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:25.006471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:34485 (local address 127.4.221.126:35055)
0807 02:24:25.012267 (+ 5796us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:25.012273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:25.012290 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:24:25.012339 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:25.012342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:25.012396 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:25.014080 (+ 1684us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:25.014200 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:25.014959 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:25.016331 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:25.016594 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:25.016701 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:25.019974 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:25.019977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:25.019979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:25.020001 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:25.021556 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:25.022228 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:25.022232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:25.022234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:25.022336 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:25.022731 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:25.022734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:25.022736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:25.022951 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:25.023044 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:25.025508 (+ 2464us) server_negotiation.cc:299] Negotiation successful
0807 02:24:25.025647 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5690,"thread_start_us":90,"threads_started":1}
I20260807 02:24:25.027637 17825 heartbeater.cc:347] Connected to a master server at 127.4.221.126:35055
I20260807 02:24:25.027897 17825 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:25.028507 17825 heartbeater.cc:511] Master 127.4.221.126:35055 requested a full tablet report, sending...
I20260807 02:24:25.030349 17607 ts_manager.cc:194] Registered new tserver with Master: eb2fef31a22343e3a41fb1d6079b4336 (127.4.221.65:45017)
I20260807 02:24:25.031884 17607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:34485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:24:25.053318 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.221.84:53741/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:35055
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:25.209610 17831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:25.210026 17831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:25.210098 17831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:25.216117 17831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:25.216200 17831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:25.216287 17831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:24:25.223317 17831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.221.84:53741/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:35055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:25.224522 17831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:25.225566 17831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:25.237502 17837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:25.237900 17840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:25.238224 17838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:25.238420 17831 server_base.cc:1034] running on GCE node
I20260807 02:24:25.238984 17831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:25.239807 17831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:25.241022 17831 hybrid_clock.cc:648] HybridClock initialized: now 1786069465241003 us; error 34 us; skew 500 ppm
Aug 07 02:24:25 dist-test-slave-69rg krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069465, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:25.244664 17831 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:24:25.245718 17831 webserver.cc:460] Webserver started at http://127.4.221.66:45887/ using document root <none> and password file <none>
I20260807 02:24:25.246331 17831 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:25.246426 17831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:25.246671 17831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:24:25 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069465, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:24:25.265005 17831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "ad8a12cafc9744a88568c59431d540db"
format_stamp: "Formatted at 2026-08-07 02:24:25 on dist-test-slave-69rg"
server_key: "e9472dfe0c495b065fb92ca3cc7e7555"
server_key_iv: "4777c7f1da82ab729ada68e6fbbe2ce3"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:25.265774 17831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "ad8a12cafc9744a88568c59431d540db"
format_stamp: "Formatted at 2026-08-07 02:24:25 on dist-test-slave-69rg"
server_key: "e9472dfe0c495b065fb92ca3cc7e7555"
server_key_iv: "4777c7f1da82ab729ada68e6fbbe2ce3"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:25.270916 17831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 02:24:25.286693 17847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:25.288134 17831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:24:25.288292 17831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "ad8a12cafc9744a88568c59431d540db"
format_stamp: "Formatted at 2026-08-07 02:24:25 on dist-test-slave-69rg"
server_key: "e9472dfe0c495b065fb92ca3cc7e7555"
server_key_iv: "4777c7f1da82ab729ada68e6fbbe2ce3"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:25.288444 17831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:25.327991 17831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:25.331665 17831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:25.331899 17831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:25.332612 17831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:25.333556 17831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:24:25.333628 17831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:25.333757 17831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:24:25.333803 17831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:25.346549 17831 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:44525
I20260807 02:24:25.347632 17831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:24:25.348459 17963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:44525 every 8 connection(s)
I20260807 02:24:25.357547 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 17831
I20260807 02:24:25.357687 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:24:25.358067 4981 external_mini_cluster.cc:1428] Setting key c36d07d42663712c75930689e6545f7f
Aug 07 02:24:25 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069465, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:25.363937 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:25.349970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:47729 (local address 127.4.221.126:35055)
0807 02:24:25.350136 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:25.350141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:25.351012 (+ 871us) server_negotiation.cc:407] Connection header received
0807 02:24:25.352152 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:25.352156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:25.352214 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:25.352272 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:25.353972 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:25.354810 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:25.356027 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:25.356269 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:25.356382 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:25.359324 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:25.359328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:25.359331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:25.359358 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:25.361144 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:25.361901 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:25.361909 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:25.361913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:25.362009 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:25.362467 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:25.362473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:25.362478 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:25.362730 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:25.362822 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:25.363530 (+ 708us) server_negotiation.cc:299] Negotiation successful
0807 02:24:25.363685 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 02:24:25.365619 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:25.350389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:35055 (local address 127.4.221.66:47729)
0807 02:24:25.350862 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:25.350914 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:25.351892 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:25.352673 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:25.352684 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:25.353069 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:25.353747 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:25.353768 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:25.354984 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:25.354986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:25.355891 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:25.355901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:25.356433 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:25.356437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:25.356604 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:25.357293 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:25.357318 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:25.359178 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:25.361287 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:25.361295 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:25.361313 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:25.361750 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:25.362134 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:25.362139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:25.362144 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:25.362341 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:25.362896 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:25.362905 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:25.363232 (+ 327us) client_negotiation.cc:769] Sending connection context
0807 02:24:25.364164 (+ 932us) client_negotiation.cc:240] Negotiation successful
0807 02:24:25.364513 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":138,"threads_started":1}
I20260807 02:24:25.366886 17964 heartbeater.cc:347] Connected to a master server at 127.4.221.126:35055
I20260807 02:24:25.367120 17964 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:25.367622 17964 heartbeater.cc:511] Master 127.4.221.126:35055 requested a full tablet report, sending...
I20260807 02:24:25.368702 17607 ts_manager.cc:194] Registered new tserver with Master: ad8a12cafc9744a88568c59431d540db (127.4.221.66:44525)
I20260807 02:24:25.369459 17607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:47729
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:24:25.401425 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.221.84:53741/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:35055
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:24:25 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069465, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
W20260807 02:24:25.568673 17969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:25.569075 17969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:25.569144 17969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:25.575745 17969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:25.575825 17969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:25.575918 17969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:24:25.584371 17969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.221.84:53741/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:35055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:25.585783 17969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:25.587216 17969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:25.599911 17976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:25.599906 17978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:25.600132 17974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:25.601328 17969 server_base.cc:1034] running on GCE node
I20260807 02:24:25.601809 17969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:25.602655 17969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:25.603856 17969 hybrid_clock.cc:648] HybridClock initialized: now 1786069465603833 us; error 38 us; skew 500 ppm
Aug 07 02:24:25 dist-test-slave-69rg krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069465, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:25.607524 17969 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:24:25.608621 17969 webserver.cc:460] Webserver started at http://127.4.221.67:46859/ using document root <none> and password file <none>
I20260807 02:24:25.609337 17969 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:25.609424 17969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:25.609666 17969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 07 02:24:25 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069465, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:24:25.634795 17969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "10d2c48788a243cfaec9bc3aea1c156f"
format_stamp: "Formatted at 2026-08-07 02:24:25 on dist-test-slave-69rg"
server_key: "c0006a720d7e8c8433b6e233272e713a"
server_key_iv: "9a719c35736c46b18111623aa92ba6d9"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:25.635941 17969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "10d2c48788a243cfaec9bc3aea1c156f"
format_stamp: "Formatted at 2026-08-07 02:24:25 on dist-test-slave-69rg"
server_key: "c0006a720d7e8c8433b6e233272e713a"
server_key_iv: "9a719c35736c46b18111623aa92ba6d9"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:25.641433 17969 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260807 02:24:25.660223 17985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:25.661985 17969 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:24:25.662155 17969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "10d2c48788a243cfaec9bc3aea1c156f"
format_stamp: "Formatted at 2026-08-07 02:24:25 on dist-test-slave-69rg"
server_key: "c0006a720d7e8c8433b6e233272e713a"
server_key_iv: "9a719c35736c46b18111623aa92ba6d9"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:25.662328 17969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:25.683127 17969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:25.686622 17969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:25.686836 17969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:25.687558 17969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:25.688432 17969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:24:25.688525 17969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:25.688592 17969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:24:25.688627 17969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:25.700021 17969 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:41597
I20260807 02:24:25.700911 17969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:24:25.702061 18101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:41597 every 8 connection(s)
I20260807 02:24:25.702369 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 17969
I20260807 02:24:25.702472 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:24:25.702714 4981 external_mini_cluster.cc:1428] Setting key ea2a40582754a6ae199cc8190d045b10
Aug 07 02:24:25 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069465, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:25.716461 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:25.705313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:39109 (local address 127.4.221.126:35055)
0807 02:24:25.705434 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:25.705438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:25.705668 (+ 230us) server_negotiation.cc:407] Connection header received
0807 02:24:25.706610 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:25.706614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:25.706668 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:25.706777 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:25.707818 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:25.708343 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:25.709477 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:25.709698 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:25.709833 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:25.712197 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:25.712203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:25.712210 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:25.712233 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:25.713969 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:25.714604 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:25.714608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:25.714610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:25.714703 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:25.715132 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:25.715136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:25.715138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:25.715376 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:25.715457 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:25.716154 (+ 697us) server_negotiation.cc:299] Negotiation successful
0807 02:24:25.716295 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 02:24:25.718057 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:25.704332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:35055 (local address 127.4.221.67:39109)
0807 02:24:25.705534 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:25.705574 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:25.706418 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:25.706944 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:25.706952 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:25.707209 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:25.707674 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:25.707687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:25.708463 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:25.708466 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:25.709344 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:25.709354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:25.709877 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:25.709881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:25.710108 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:25.710707 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:25.710723 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:25.712042 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:25.714105 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:25.714111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:25.714125 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:25.714480 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:25.714827 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:25.714829 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:25.714831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:25.715010 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:25.715551 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:25.715557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:25.715859 (+ 302us) client_negotiation.cc:769] Sending connection context
0807 02:24:25.716677 (+ 818us) client_negotiation.cc:240] Negotiation successful
0807 02:24:25.717021 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"thread_start_us":136,"threads_started":1}
I20260807 02:24:25.719167 18102 heartbeater.cc:347] Connected to a master server at 127.4.221.126:35055
I20260807 02:24:25.719409 18102 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:25.719936 18102 heartbeater.cc:511] Master 127.4.221.126:35055 requested a full tablet report, sending...
I20260807 02:24:25.720973 17607 ts_manager.cc:194] Registered new tserver with Master: 10d2c48788a243cfaec9bc3aea1c156f (127.4.221.67:41597)
I20260807 02:24:25.721704 17607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:39109
I20260807 02:24:25.729617 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:24:25.730758 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 17690
I20260807 02:24:25.747370 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 17831
I20260807 02:24:25.755074 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 17969
I20260807 02:24:25.766201 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 17574
I20260807 02:24:25.774382 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:35055
--webserver_interface=127.4.221.126
--webserver_port=44209
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:35055
--ranger_config_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.221.84:53741/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.221.84:53741/kms with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:25.895239 18106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:25.895526 18106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:25.895571 18106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:25.900002 18106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:24:25.900063 18106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:25.900085 18106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:25.900103 18106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:24:25.900120 18106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:24:25.906132 18106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.221.84:53741/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:35055
--ranger_config_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:35055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=44209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:25.907159 18106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:25.908166 18106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:25.917171 18111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:25.917191 18112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:25.917331 18114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:25.917280 18106 server_base.cc:1034] running on GCE node
I20260807 02:24:25.918093 18106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:25.919174 18106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:25.920342 18106 hybrid_clock.cc:648] HybridClock initialized: now 1786069465920315 us; error 39 us; skew 500 ppm
Aug 07 02:24:25 dist-test-slave-69rg krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069465, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:25.923221 18106 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:24:25.924026 18106 webserver.cc:460] Webserver started at http://127.4.221.126:44209/ using document root <none> and password file <none>
I20260807 02:24:25.924518 18106 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:25.924583 18106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 02:24:25 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069465, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:24:26.005307 18106 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:24:26.007684 18122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:26.008961 18106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:24:26.009121 18106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "7f5483b891c645aebabee96557c6d391"
format_stamp: "Formatted at 2026-08-07 02:24:23 on dist-test-slave-69rg"
server_key: "d7fb51cd33d5a6330ca39fb2b8fa4a61"
server_key_iv: "1c1025525af7044b6fb3b18e3e24635f"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:26.009675 18106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:26.020689 18106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:26.023942 18106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:26.024156 18106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:26.031678 18106 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:35055
I20260807 02:24:26.031766 18177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:35055 every 8 connection(s)
I20260807 02:24:26.032727 18106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:24:26.033022 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 18106
I20260807 02:24:26.033553 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.221.84:53741/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:45017
--local_ip_for_outbound_sockets=127.4.221.65
--tserver_master_addrs=127.4.221.126:35055
--webserver_port=46747
--webserver_interface=127.4.221.65
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.221.84:53741/kms with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:24:26.037667 18178 sys_catalog.cc:263] Verifying existing consensus state
I20260807 02:24:26.039392 18178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391: Bootstrap starting.
I20260807 02:24:26.055794 18178 log.cc:824] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:26.063952 18178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:24:26.064484 18178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391: Bootstrap complete.
I20260807 02:24:26.069042 18178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5483b891c645aebabee96557c6d391" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35055 } }
I20260807 02:24:26.069466 18178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5483b891c645aebabee96557c6d391, State: Initialized, Role: FOLLOWER
I20260807 02:24:26.069980 18178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5483b891c645aebabee96557c6d391" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35055 } }
I20260807 02:24:26.070124 18178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:24:26.070214 18178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:24:26.070312 18178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 1 FOLLOWER]: Advancing to term 2
I20260807 02:24:26.072294 18178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5483b891c645aebabee96557c6d391" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35055 } }
I20260807 02:24:26.072618 18178 leader_election.cc:302] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f5483b891c645aebabee96557c6d391; no voters:
I20260807 02:24:26.072942 18178 leader_election.cc:288] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 02:24:26.073074 18182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 02:24:26.073340 18182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [term 2 LEADER]: Becoming Leader. State: Replica: 7f5483b891c645aebabee96557c6d391, State: Running, Role: LEADER
I20260807 02:24:26.073699 18182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5483b891c645aebabee96557c6d391" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35055 } }
I20260807 02:24:26.074095 18178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:24:26.074645 18182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f5483b891c645aebabee96557c6d391. Latest consensus state: current_term: 2 leader_uuid: "7f5483b891c645aebabee96557c6d391" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5483b891c645aebabee96557c6d391" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35055 } } }
I20260807 02:24:26.074761 18182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:26.075239 18183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7f5483b891c645aebabee96557c6d391" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5483b891c645aebabee96557c6d391" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 35055 } } }
I20260807 02:24:26.075387 18183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391 [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:26.076735 18178 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260807 02:24:26.077026 18187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:24:26.083112 18187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:24:26.084069 18187 catalog_manager.cc:1259] Loaded cluster ID: 350d2cb30d2b48da91edc808e03b5235
I20260807 02:24:26.084197 18187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:24:26.087188 18187 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:24:26.088083 18187 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391: Loaded TSK: 0
I20260807 02:24:26.089557 18187 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 02:24:26.177471 18180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:26.177862 18180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:26.177937 18180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:26.183924 18180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:26.184046 18180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:26.184175 18180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:24:26.191902 18180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.221.84:53741/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:45017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=46747
--tserver_master_addrs=127.4.221.126:35055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:26.193413 18180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:26.194733 18180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:26.206696 18218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:26.206693 18216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:26.207753 18215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:26.207994 18180 server_base.cc:1034] running on GCE node
I20260807 02:24:26.209010 18180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:26.209776 18180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:26.210964 18180 hybrid_clock.cc:648] HybridClock initialized: now 1786069466210941 us; error 38 us; skew 500 ppm
Aug 07 02:24:26 dist-test-slave-69rg krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:26.214638 18180 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:24:26.215698 18180 webserver.cc:460] Webserver started at http://127.4.221.65:46747/ using document root <none> and password file <none>
I20260807 02:24:26.216358 18180 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:26.216454 18180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 02:24:26 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:24:26.239514 18180 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.001s
I20260807 02:24:26.242383 18227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:26.243942 18180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:24:26.244081 18180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "eb2fef31a22343e3a41fb1d6079b4336"
format_stamp: "Formatted at 2026-08-07 02:24:24 on dist-test-slave-69rg"
server_key: "f2250b8f986fcf98937eeb379878b326"
server_key_iv: "b0599d96f064c3eca168f37184d735c3"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:26.244831 18180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:26.265036 18180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:26.268451 18180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:26.268666 18180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:26.269323 18180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:26.270406 18180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:24:26.270468 18180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:26.270531 18180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:24:26.270558 18180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:26.284014 18180 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:45017
I20260807 02:24:26.285199 18180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:24:26.286798 18345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:45017 every 8 connection(s)
I20260807 02:24:26.292364 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 18180
I20260807 02:24:26.292876 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.221.84:53741/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:44525
--local_ip_for_outbound_sockets=127.4.221.66
--tserver_master_addrs=127.4.221.126:35055
--webserver_port=45887
--webserver_interface=127.4.221.66
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.221.84:53741/kms with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:24:26 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:26.316743 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:26.288882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:35055 (local address 127.4.221.65:53329)
0807 02:24:26.291707 (+ 2825us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:26.291765 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:26.292780 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:26.294526 (+ 1746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:26.294533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:26.294863 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:26.295433 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:26.295450 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:26.297243 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:26.297249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:26.298604 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:26.298615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:26.299957 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:26.299964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:26.300204 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:26.301097 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:26.301139 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:26.307537 (+ 6398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:26.312948 (+ 5411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:26.312955 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:26.312971 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:26.313342 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:26.313759 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:26.313762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:26.313764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:26.314009 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:26.314810 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:26.314816 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:26.315108 (+ 292us) client_negotiation.cc:769] Sending connection context
0807 02:24:26.315308 (+ 200us) client_negotiation.cc:240] Negotiation successful
0807 02:24:26.315656 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2569,"thread_start_us":125,"threads_started":1}
I20260807 02:24:26.318473 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:26.290282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:53329 (local address 127.4.221.126:35055)
0807 02:24:26.291348 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:26.291376 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:26.293099 (+ 1723us) server_negotiation.cc:407] Connection header received
0807 02:24:26.293378 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:26.293391 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:26.293754 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:26.294092 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:26.295630 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:26.297071 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:26.298759 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:26.299517 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:26.299707 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:26.307872 (+ 8165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:26.307883 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:26.307906 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:26.307950 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:26.312751 (+ 4801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:26.313491 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:26.313498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:26.313503 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:26.313649 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:26.314116 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:26.314129 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:26.314131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:26.314627 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:26.314754 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:26.317295 (+ 2541us) server_negotiation.cc:299] Negotiation successful
0807 02:24:26.317578 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":899,"thread_start_us":102,"threads_started":1}
I20260807 02:24:26.323190 18346 heartbeater.cc:347] Connected to a master server at 127.4.221.126:35055
I20260807 02:24:26.323432 18346 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:26.323976 18346 heartbeater.cc:511] Master 127.4.221.126:35055 requested a full tablet report, sending...
W20260807 02:24:26.326321 18346 heartbeater.cc:644] Failed to heartbeat to 127.4.221.126:35055 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 02:24:26.326460 18346 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:26.326689 18346 heartbeater.cc:511] Master 127.4.221.126:35055 requested a full tablet report, sending...
I20260807 02:24:26.328800 18346 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:26.328977 18346 heartbeater.cc:511] Master 127.4.221.126:35055 requested a full tablet report, sending...
W20260807 02:24:26.329728 18346 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 02:24:26.430600 18350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:26.430913 18350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:26.430979 18350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:26.435211 18350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:26.435284 18350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:26.435395 18350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:24:26.441311 18350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.221.84:53741/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:44525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=45887
--tserver_master_addrs=127.4.221.126:35055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:26.442386 18350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:26.443387 18350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:26.454001 18355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:26.454016 18358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:26.454129 18356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:26.454924 18350 server_base.cc:1034] running on GCE node
I20260807 02:24:26.455355 18350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:26.456112 18350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:26.457298 18350 hybrid_clock.cc:648] HybridClock initialized: now 1786069466457283 us; error 33 us; skew 500 ppm
Aug 07 02:24:26 dist-test-slave-69rg krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:26.460949 18350 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:24:26.461934 18350 webserver.cc:460] Webserver started at http://127.4.221.66:45887/ using document root <none> and password file <none>
I20260807 02:24:26.462481 18350 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:26.462584 18350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 02:24:26 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:24:26.487644 18350 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 02:24:26.491183 18366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:26.492652 18350 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260807 02:24:26.492779 18350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "ad8a12cafc9744a88568c59431d540db"
format_stamp: "Formatted at 2026-08-07 02:24:25 on dist-test-slave-69rg"
server_key: "e9472dfe0c495b065fb92ca3cc7e7555"
server_key_iv: "4777c7f1da82ab729ada68e6fbbe2ce3"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:26.493489 18350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:26.522181 18350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:26.525938 18350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:26.526146 18350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:26.526916 18350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:26.527964 18350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:24:26.528031 18350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:26.528090 18350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:24:26.528115 18350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:26.540639 18350 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:44525
I20260807 02:24:26.542482 18486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:44525 every 8 connection(s)
I20260807 02:24:26.542785 18350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:24:26.545267 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 18350
I20260807 02:24:26.545769 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.221.84:53741/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:41597
--local_ip_for_outbound_sockets=127.4.221.67
--tserver_master_addrs=127.4.221.126:35055
--webserver_port=46859
--webserver_interface=127.4.221.67
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.221.84:53741/kms with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:24:26 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:26.560112 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:26.546528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:45531 (local address 127.4.221.126:35055)
0807 02:24:26.546676 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:26.546680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:26.547119 (+ 439us) server_negotiation.cc:407] Connection header received
0807 02:24:26.548370 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:26.548374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:26.548436 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:26.548494 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:26.550170 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:26.550971 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:26.552722 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:26.553073 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:26.553201 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:26.556269 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:26.556272 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:26.556275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:26.556310 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:26.557990 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:26.558510 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:26.558513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:26.558515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:26.558605 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:26.558961 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:26.558964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:26.558966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:26.559153 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:26.559272 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:26.559800 (+ 528us) server_negotiation.cc:299] Negotiation successful
0807 02:24:26.559934 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:24:26.561117 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:26.545377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:35055 (local address 127.4.221.66:45531)
0807 02:24:26.546945 (+ 1568us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:26.547004 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:26.548100 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:26.548927 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:26.548936 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:26.549307 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:26.549979 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:26.550000 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:26.551144 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:26.551147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:26.552565 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:26.552576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:26.553381 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:26.553386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:26.553590 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:26.554313 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:26.554335 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:26.556080 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:26.558104 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:26.558121 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:26.558135 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:26.558396 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:26.558702 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:26.558706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:26.558708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:26.558853 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:26.559267 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:26.559273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:26.559522 (+ 249us) client_negotiation.cc:769] Sending connection context
0807 02:24:26.559742 (+ 220us) client_negotiation.cc:240] Negotiation successful
0807 02:24:26.560060 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1350,"thread_start_us":148,"threads_started":1}
I20260807 02:24:26.562484 18487 heartbeater.cc:347] Connected to a master server at 127.4.221.126:35055
I20260807 02:24:26.562750 18487 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:26.563318 18487 heartbeater.cc:511] Master 127.4.221.126:35055 requested a full tablet report, sending...
W20260807 02:24:26.564965 18487 heartbeater.cc:644] Failed to heartbeat to 127.4.221.126:35055 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 02:24:26.565097 18487 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:26.565274 18487 heartbeater.cc:511] Master 127.4.221.126:35055 requested a full tablet report, sending...
I20260807 02:24:26.566244 18487 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:26.566411 18487 heartbeater.cc:511] Master 127.4.221.126:35055 requested a full tablet report, sending...
W20260807 02:24:26.568147 18487 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260807 02:24:26.694401 18491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:26.694684 18491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:26.694761 18491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:26.699352 18491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:26.699429 18491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:26.699514 18491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:24:26.706800 18491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.221.84:53741/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:41597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=46859
--tserver_master_addrs=127.4.221.126:35055
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:26.708057 18491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:26.709460 18491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:26.722004 18497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:26.722620 18500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:26.722649 18491 server_base.cc:1034] running on GCE node
W20260807 02:24:26.722051 18498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:26.725070 18491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:26.725885 18491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:26.727097 18491 hybrid_clock.cc:648] HybridClock initialized: now 1786069466727056 us; error 53 us; skew 500 ppm
Aug 07 02:24:26 dist-test-slave-69rg krb5kdc[17117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:26.730708 18491 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:24:26.731806 18491 webserver.cc:460] Webserver started at http://127.4.221.67:46859/ using document root <none> and password file <none>
I20260807 02:24:26.732498 18491 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:26.732607 18491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 07 02:24:26 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:24:26.751319 18491 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260807 02:24:26.754364 18508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:26.755963 18491 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260807 02:24:26.756223 18491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "10d2c48788a243cfaec9bc3aea1c156f"
format_stamp: "Formatted at 2026-08-07 02:24:25 on dist-test-slave-69rg"
server_key: "c0006a720d7e8c8433b6e233272e713a"
server_key_iv: "9a719c35736c46b18111623aa92ba6d9"
server_key_version: "kuduclusterkey@0"
I20260807 02:24:26.756961 18491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:26.772073 18491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:26.775656 18491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:26.775882 18491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:26.776628 18491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:26.777654 18491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:24:26.777719 18491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:26.777794 18491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:24:26.777848 18491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260807 02:24:26.790429 18491 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:41597
I20260807 02:24:26.791838 18491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:24:26.793167 18624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:41597 every 8 connection(s)
I20260807 02:24:26.797067 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 18491
Aug 07 02:24:26 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069466, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:26.807515 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:26.794096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:60391 (local address 127.4.221.126:35055)
0807 02:24:26.794216 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:26.794221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:26.795395 (+ 1174us) server_negotiation.cc:407] Connection header received
0807 02:24:26.796447 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:26.796451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:26.796510 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:26.796586 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:26.798452 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:26.799099 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:26.800674 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:26.800997 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:26.801111 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:26.803677 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:26.803681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:26.803683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:26.803707 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:26.805285 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:26.805779 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:26.805783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:26.805785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:26.805922 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:26.806317 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:26.806320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:26.806322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:26.806521 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:26.806607 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:26.807233 (+ 626us) server_negotiation.cc:299] Negotiation successful
0807 02:24:26.807348 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260807 02:24:26.808460 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:26.794732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:35055 (local address 127.4.221.67:60391)
0807 02:24:26.795250 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:26.795290 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:26.796231 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:26.796804 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:26.796813 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:26.797241 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:26.798308 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:26.798322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:26.799218 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:26.799221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:26.800516 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:26.800528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:26.801176 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:26.801183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:26.801380 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:26.802069 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:26.802089 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:26.803513 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:26.805405 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:26.805410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:26.805423 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:26.805671 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:26.806034 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:26.806037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:26.806039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:26.806206 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:26.806670 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:26.806676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:26.806952 (+ 276us) client_negotiation.cc:769] Sending connection context
0807 02:24:26.807166 (+ 214us) client_negotiation.cc:240] Negotiation successful
0807 02:24:26.807468 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":153,"threads_started":1}
I20260807 02:24:26.809599 18625 heartbeater.cc:347] Connected to a master server at 127.4.221.126:35055
I20260807 02:24:26.809837 18625 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:26.810371 18625 heartbeater.cc:511] Master 127.4.221.126:35055 requested a full tablet report, sending...
W20260807 02:24:26.811910 18625 heartbeater.cc:644] Failed to heartbeat to 127.4.221.126:35055 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260807 02:24:26.812039 18625 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:26.812238 18625 heartbeater.cc:511] Master 127.4.221.126:35055 requested a full tablet report, sending...
I20260807 02:24:26.813099 18625 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:26.813601 18625 heartbeater.cc:511] Master 127.4.221.126:35055 requested a full tablet report, sending...
W20260807 02:24:26.814783 18625 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 02:24:26.817159 18630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:26.797739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46240 (local address 127.4.221.126:35055)
0807 02:24:26.798561 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:26.798565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:26.798576 (+ 11us) server_negotiation.cc:407] Connection header received
0807 02:24:26.798604 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:26.798606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:26.798657 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:26.798740 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:26.801220 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:26.802309 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:26.809841 (+ 7532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:26.810162 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:26.810263 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:26.812893 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:26.812897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:26.812900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:26.812922 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:26.814927 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:26.815787 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:26.815790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:26.815792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:26.815877 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:26.816185 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:26.816189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:26.816190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:26.816365 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:26.816566 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:26.816753 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 02:24:26.816867 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":128,"threads_started":1}
Aug 07 02:24:27 dist-test-slave-69rg krb5kdc[17117](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786069467, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:27.329964 18346 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:27.330327 18346 heartbeater.cc:511] Master 127.4.221.126:35055 requested a full tablet report, sending...
I20260807 02:24:27.334233 18139 ts_manager.cc:194] Registered new tserver with Master: eb2fef31a22343e3a41fb1d6079b4336 (127.4.221.65:45017)
I20260807 02:24:27.335971 18139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:53329
Aug 07 02:24:27 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069467, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:24:27.568361 18487 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:27.568667 18487 heartbeater.cc:511] Master 127.4.221.126:35055 requested a full tablet report, sending...
I20260807 02:24:27.569432 18138 ts_manager.cc:194] Registered new tserver with Master: ad8a12cafc9744a88568c59431d540db (127.4.221.66:44525)
I20260807 02:24:27.570133 18138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:45531
Aug 07 02:24:27 dist-test-slave-69rg krb5kdc[17117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069467, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for HTTP/127.4.221.84@KRBTEST.COM
I20260807 02:24:27.815013 18625 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:27.815315 18625 heartbeater.cc:511] Master 127.4.221.126:35055 requested a full tablet report, sending...
I20260807 02:24:27.816077 18138 ts_manager.cc:194] Registered new tserver with Master: 10d2c48788a243cfaec9bc3aea1c156f (127.4.221.67:41597)
I20260807 02:24:27.816898 18138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:60391
I20260807 02:24:27.826824 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:24:27.848837 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:27.834309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46244 (local address 127.4.221.126:35055)
0807 02:24:27.838024 (+ 3715us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:27.838031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:27.838483 (+ 452us) server_negotiation.cc:407] Connection header received
0807 02:24:27.838539 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:27.838542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:27.838593 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:27.838682 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:27.839723 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:27.840569 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:27.842161 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:27.842529 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:27.842638 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:27.844595 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:27.844600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:27.844602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:27.844634 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:27.846589 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:27.847140 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:27.847144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:27.847146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:27.847246 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:27.847590 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:27.847594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:27.847595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:27.847798 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:27.847954 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:27.848570 (+ 616us) server_negotiation.cc:299] Negotiation successful
0807 02:24:27.848689 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3617,"thread_start_us":79,"threads_started":1}
I20260807 02:24:27.853861 18138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:24:27.859145 18138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:24:27.882786 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:27.876079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:45017 (local address 127.0.0.1:53810)
0807 02:24:27.877984 (+ 1905us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:27.877999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:27.878105 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:27.878551 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:27.878557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:27.878579 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:27.878782 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:27.878798 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:27.879875 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:27.879879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:27.881199 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:27.881209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:27.882243 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:27.882246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:27.882362 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:27.882397 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 02:24:27.882503 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 02:24:27.882614 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1833,"thread_start_us":122,"threads_started":1}
I20260807 02:24:27.884155 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:27.876256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.4.221.65:45017)
0807 02:24:27.876576 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:27.876585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:27.878099 (+ 1514us) server_negotiation.cc:407] Connection header received
0807 02:24:27.878226 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:27.878230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:27.878392 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:27.878558 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:27.878942 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:27.879750 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:27.881326 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:27.881942 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:27.882042 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:27.882182 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:27.883898 (+ 1716us) server_negotiation.cc:299] Negotiation successful
0807 02:24:27.884007 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":101,"threads_started":1}
I20260807 02:24:27.887784 18276 tablet_service.cc:1467] Processing CreateTablet for tablet b5feb08020fe4cfda96a77407f2eb7e8 (DEFAULT_TABLE table=test-table [id=4d9886cfa9214dbcbcbd73fd181a8be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:24:27.888993 18276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5feb08020fe4cfda96a77407f2eb7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:27.890936 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:27.876345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:44525 (local address 127.0.0.1:36288)
0807 02:24:27.884808 (+ 8463us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:27.884822 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:27.884983 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:27.885732 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:27.885735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:27.885754 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:27.885863 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:27.885870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:27.888072 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:27.888075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:27.889193 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:27.889206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:27.890514 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:27.890518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:27.890611 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:27.890649 (+ 38us) client_negotiation.cc:769] Sending connection context
0807 02:24:27.890722 (+ 73us) client_negotiation.cc:240] Negotiation successful
0807 02:24:27.890789 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8406,"thread_start_us":89,"threads_started":1}
I20260807 02:24:27.891355 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:27.876708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36288 (local address 127.4.221.66:44525)
0807 02:24:27.877950 (+ 1242us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:27.877955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:27.884903 (+ 6948us) server_negotiation.cc:407] Connection header received
0807 02:24:27.885138 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:27.885145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:27.885373 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:27.885503 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:27.886017 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:27.887938 (+ 1921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:27.889337 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:27.890200 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:27.890308 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:27.890450 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:27.891121 (+ 671us) server_negotiation.cc:299] Negotiation successful
0807 02:24:27.891219 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1137,"thread_start_us":75,"threads_started":1}
I20260807 02:24:27.895426 18416 tablet_service.cc:1467] Processing CreateTablet for tablet b5feb08020fe4cfda96a77407f2eb7e8 (DEFAULT_TABLE table=test-table [id=4d9886cfa9214dbcbcbd73fd181a8be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:24:27.896487 18416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5feb08020fe4cfda96a77407f2eb7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:27.904402 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:27.873433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:41597 (local address 127.0.0.1:49306)
0807 02:24:27.877100 (+ 3667us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:27.877153 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:27.877428 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:27.897885 (+ 20457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:27.897889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:27.897910 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:27.898012 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:27.898019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:27.899773 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:27.899777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:27.900964 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:27.900976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:27.903960 (+ 2984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:27.903965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:27.904065 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:27.904095 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 02:24:27.904150 (+ 55us) client_negotiation.cc:240] Negotiation successful
0807 02:24:27.904238 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3461,"thread_start_us":111,"threads_started":1}
I20260807 02:24:27.905351 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:27.874463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49306 (local address 127.4.221.67:41597)
0807 02:24:27.897346 (+ 22883us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:27.897353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:27.897384 (+ 31us) server_negotiation.cc:407] Connection header received
0807 02:24:27.897467 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:27.897473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:27.897675 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:27.898213 (+ 538us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:27.898363 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:27.899512 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:27.901127 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:27.903810 (+ 2683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:27.904889 (+ 1079us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:27.905045 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:27.905098 (+ 53us) server_negotiation.cc:299] Negotiation successful
0807 02:24:27.905207 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10115,"thread_start_us":96,"threads_started":1}
I20260807 02:24:27.908591 18554 tablet_service.cc:1467] Processing CreateTablet for tablet b5feb08020fe4cfda96a77407f2eb7e8 (DEFAULT_TABLE table=test-table [id=4d9886cfa9214dbcbcbd73fd181a8be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:24:27.909724 18554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5feb08020fe4cfda96a77407f2eb7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:27.913107 18663 tablet_bootstrap.cc:492] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336: Bootstrap starting.
I20260807 02:24:27.915804 18663 tablet_bootstrap.cc:654] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:27.916774 18663 log.cc:824] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:27.921672 18666 tablet_bootstrap.cc:492] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f: Bootstrap starting.
I20260807 02:24:27.922147 18664 tablet_bootstrap.cc:492] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db: Bootstrap starting.
I20260807 02:24:27.927912 18666 tablet_bootstrap.cc:654] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:27.928882 18666 log.cc:824] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:27.929090 18664 tablet_bootstrap.cc:654] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:27.929961 18664 log.cc:824] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:27.931818 18663 tablet_bootstrap.cc:492] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336: No bootstrap required, opened a new log
I20260807 02:24:27.932088 18663 ts_tablet_manager.cc:1397] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260807 02:24:27.932802 18666 tablet_bootstrap.cc:492] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f: No bootstrap required, opened a new log
I20260807 02:24:27.932986 18666 ts_tablet_manager.cc:1397] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260807 02:24:27.937054 18663 raft_consensus.cc:348] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d2c48788a243cfaec9bc3aea1c156f" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41597 } } peers { permanent_uuid: "eb2fef31a22343e3a41fb1d6079b4336" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45017 } } peers { permanent_uuid: "ad8a12cafc9744a88568c59431d540db" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 44525 } }
I20260807 02:24:27.937461 18663 raft_consensus.cc:374] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:27.937600 18663 raft_consensus.cc:729] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2fef31a22343e3a41fb1d6079b4336, State: Initialized, Role: FOLLOWER
I20260807 02:24:27.938227 18663 consensus_queue.cc:260] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d2c48788a243cfaec9bc3aea1c156f" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41597 } } peers { permanent_uuid: "eb2fef31a22343e3a41fb1d6079b4336" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45017 } } peers { permanent_uuid: "ad8a12cafc9744a88568c59431d540db" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 44525 } }
I20260807 02:24:27.938144 18666 raft_consensus.cc:348] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d2c48788a243cfaec9bc3aea1c156f" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41597 } } peers { permanent_uuid: "eb2fef31a22343e3a41fb1d6079b4336" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45017 } } peers { permanent_uuid: "ad8a12cafc9744a88568c59431d540db" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 44525 } }
I20260807 02:24:27.938390 18666 raft_consensus.cc:374] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:27.938452 18666 raft_consensus.cc:729] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10d2c48788a243cfaec9bc3aea1c156f, State: Initialized, Role: FOLLOWER
I20260807 02:24:27.938881 18666 consensus_queue.cc:260] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d2c48788a243cfaec9bc3aea1c156f" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41597 } } peers { permanent_uuid: "eb2fef31a22343e3a41fb1d6079b4336" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45017 } } peers { permanent_uuid: "ad8a12cafc9744a88568c59431d540db" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 44525 } }
I20260807 02:24:27.939585 18663 ts_tablet_manager.cc:1428] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 02:24:27.939852 18666 ts_tablet_manager.cc:1428] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 02:24:27.939996 18346 heartbeater.cc:503] Master 127.4.221.126:35055 was elected leader, sending a full tablet report...
I20260807 02:24:27.941540 18664 tablet_bootstrap.cc:492] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db: No bootstrap required, opened a new log
I20260807 02:24:27.941754 18664 ts_tablet_manager.cc:1397] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20260807 02:24:27.946044 18664 raft_consensus.cc:348] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d2c48788a243cfaec9bc3aea1c156f" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41597 } } peers { permanent_uuid: "eb2fef31a22343e3a41fb1d6079b4336" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45017 } } peers { permanent_uuid: "ad8a12cafc9744a88568c59431d540db" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 44525 } }
I20260807 02:24:27.946310 18664 raft_consensus.cc:374] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:27.946384 18664 raft_consensus.cc:729] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad8a12cafc9744a88568c59431d540db, State: Initialized, Role: FOLLOWER
I20260807 02:24:27.946897 18664 consensus_queue.cc:260] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d2c48788a243cfaec9bc3aea1c156f" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41597 } } peers { permanent_uuid: "eb2fef31a22343e3a41fb1d6079b4336" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45017 } } peers { permanent_uuid: "ad8a12cafc9744a88568c59431d540db" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 44525 } }
I20260807 02:24:27.947903 18664 ts_tablet_manager.cc:1428] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260807 02:24:27.948273 18487 heartbeater.cc:503] Master 127.4.221.126:35055 was elected leader, sending a full tablet report...
I20260807 02:24:27.948354 18625 heartbeater.cc:503] Master 127.4.221.126:35055 was elected leader, sending a full tablet report...
W20260807 02:24:28.043300 18347 tablet.cc:2369] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:24:28.044900 18626 tablet.cc:2369] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:24:28.044991 18489 tablet.cc:2369] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:24:28.230693 18669 raft_consensus.cc:482] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:24:28.230944 18669 raft_consensus.cc:504] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d2c48788a243cfaec9bc3aea1c156f" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41597 } } peers { permanent_uuid: "eb2fef31a22343e3a41fb1d6079b4336" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45017 } } peers { permanent_uuid: "ad8a12cafc9744a88568c59431d540db" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 44525 } }
I20260807 02:24:28.232147 18669 leader_election.cc:288] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 10d2c48788a243cfaec9bc3aea1c156f (127.4.221.67:41597), ad8a12cafc9744a88568c59431d540db (127.4.221.66:44525)
I20260807 02:24:28.236578 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:28.232738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:58189 (local address 127.4.221.66:44525)
0807 02:24:28.232887 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:28.232891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:28.233069 (+ 178us) server_negotiation.cc:407] Connection header received
0807 02:24:28.233231 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:28.233234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:28.233284 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:28.233373 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:28.233653 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:28.234236 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:28.235462 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:28.236041 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:28.236170 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:28.236214 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:28.236353 (+ 139us) server_negotiation.cc:299] Negotiation successful
0807 02:24:28.236432 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 02:24:28.236604 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:28.232483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:44525 (local address 127.4.221.65:58189)
0807 02:24:28.232949 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:28.232962 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:28.233115 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:28.233395 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:28.233398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:28.233413 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:28.233544 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:28.233549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:28.234348 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:28.234355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:28.235332 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:28.235342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:28.236154 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:28.236156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:28.236236 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:28.236257 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 02:24:28.236320 (+ 63us) client_negotiation.cc:240] Negotiation successful
0807 02:24:28.236401 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":52,"thread_start_us":140,"threads_started":1}
I20260807 02:24:28.236769 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:28.232589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:45985 (local address 127.4.221.67:41597)
0807 02:24:28.232756 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:28.232763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:28.233025 (+ 262us) server_negotiation.cc:407] Connection header received
0807 02:24:28.233212 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:28.233215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:28.233256 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:28.233338 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:28.233681 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:28.234270 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:28.235456 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:28.236047 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:28.236163 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:28.236214 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:28.236560 (+ 346us) server_negotiation.cc:299] Negotiation successful
0807 02:24:28.236649 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260807 02:24:28.236999 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:28.232483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:41597 (local address 127.4.221.65:45985)
0807 02:24:28.232910 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:28.232925 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:28.233073 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:28.233359 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:28.233362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:28.233378 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:28.233565 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:28.233570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:28.234380 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:28.234386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:28.235293 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:28.235303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:28.236288 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:28.236292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:28.236417 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:28.236439 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 02:24:28.236509 (+ 70us) client_negotiation.cc:240] Negotiation successful
0807 02:24:28.236565 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"spinlock_wait_cycles":16640,"thread_start_us":136,"threads_started":1}
I20260807 02:24:28.237290 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5feb08020fe4cfda96a77407f2eb7e8" candidate_uuid: "eb2fef31a22343e3a41fb1d6079b4336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8a12cafc9744a88568c59431d540db" is_pre_election: true
I20260807 02:24:28.237607 18419 raft_consensus.cc:2454] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2fef31a22343e3a41fb1d6079b4336 in term 0.
I20260807 02:24:28.237605 18577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5feb08020fe4cfda96a77407f2eb7e8" candidate_uuid: "eb2fef31a22343e3a41fb1d6079b4336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d2c48788a243cfaec9bc3aea1c156f" is_pre_election: true
I20260807 02:24:28.237897 18577 raft_consensus.cc:2454] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2fef31a22343e3a41fb1d6079b4336 in term 0.
I20260807 02:24:28.238098 18228 leader_election.cc:302] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad8a12cafc9744a88568c59431d540db, eb2fef31a22343e3a41fb1d6079b4336; no voters:
I20260807 02:24:28.238399 18669 raft_consensus.cc:2781] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:24:28.238489 18669 raft_consensus.cc:482] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:24:28.238533 18669 raft_consensus.cc:3037] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:28.239648 18669 raft_consensus.cc:504] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d2c48788a243cfaec9bc3aea1c156f" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41597 } } peers { permanent_uuid: "eb2fef31a22343e3a41fb1d6079b4336" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45017 } } peers { permanent_uuid: "ad8a12cafc9744a88568c59431d540db" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 44525 } }
I20260807 02:24:28.239964 18669 leader_election.cc:288] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [CANDIDATE]: Term 1 election: Requested vote from peers 10d2c48788a243cfaec9bc3aea1c156f (127.4.221.67:41597), ad8a12cafc9744a88568c59431d540db (127.4.221.66:44525)
I20260807 02:24:28.240351 18577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5feb08020fe4cfda96a77407f2eb7e8" candidate_uuid: "eb2fef31a22343e3a41fb1d6079b4336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10d2c48788a243cfaec9bc3aea1c156f"
I20260807 02:24:28.240456 18577 raft_consensus.cc:3037] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:28.241441 18577 raft_consensus.cc:2454] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2fef31a22343e3a41fb1d6079b4336 in term 1.
I20260807 02:24:28.241685 18419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5feb08020fe4cfda96a77407f2eb7e8" candidate_uuid: "eb2fef31a22343e3a41fb1d6079b4336" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad8a12cafc9744a88568c59431d540db"
I20260807 02:24:28.241736 18230 leader_election.cc:302] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10d2c48788a243cfaec9bc3aea1c156f, eb2fef31a22343e3a41fb1d6079b4336; no voters:
I20260807 02:24:28.241835 18419 raft_consensus.cc:3037] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:28.241935 18669 raft_consensus.cc:2781] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:24:28.242168 18669 raft_consensus.cc:686] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [term 1 LEADER]: Becoming Leader. State: Replica: eb2fef31a22343e3a41fb1d6079b4336, State: Running, Role: LEADER
I20260807 02:24:28.242496 18669 consensus_queue.cc:237] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d2c48788a243cfaec9bc3aea1c156f" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41597 } } peers { permanent_uuid: "eb2fef31a22343e3a41fb1d6079b4336" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45017 } } peers { permanent_uuid: "ad8a12cafc9744a88568c59431d540db" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 44525 } }
I20260807 02:24:28.242985 18419 raft_consensus.cc:2454] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2fef31a22343e3a41fb1d6079b4336 in term 1.
I20260807 02:24:28.245674 18139 catalog_manager.cc:5495] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2fef31a22343e3a41fb1d6079b4336 (127.4.221.65). New cstate: current_term: 1 leader_uuid: "eb2fef31a22343e3a41fb1d6079b4336" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10d2c48788a243cfaec9bc3aea1c156f" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb2fef31a22343e3a41fb1d6079b4336" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad8a12cafc9744a88568c59431d540db" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 44525 } health_report { overall_health: UNKNOWN } } }
I20260807 02:24:28.332712 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:28.329038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.4.221.65:45017)
0807 02:24:28.329189 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:28.329194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:28.329210 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:24:28.329409 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:28.329412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:28.329469 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:28.329582 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:24:28.329851 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:28.330359 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:28.331236 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:28.331463 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:28.331577 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:28.331887 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:24:28.332288 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:24:28.332418 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:28.332481 (+ 63us) server_negotiation.cc:299] Negotiation successful
0807 02:24:28.332537 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 02:24:28.339291 18577 raft_consensus.cc:1261] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f [term 1 FOLLOWER]: Refusing update from remote peer eb2fef31a22343e3a41fb1d6079b4336: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:24:28.339349 18419 raft_consensus.cc:1261] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db [term 1 FOLLOWER]: Refusing update from remote peer eb2fef31a22343e3a41fb1d6079b4336: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:24:28.339998 18669 consensus_queue.cc:1035] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10d2c48788a243cfaec9bc3aea1c156f" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:24:28.340226 18681 consensus_queue.cc:1035] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad8a12cafc9744a88568c59431d540db" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 44525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:24:28.349637 18686 mvcc.cc:205] Tried to move back new op lower bound from 7315740542310703104 to 7315740541923934208. Current Snapshot: MvccSnapshot[applied={T|T < 7315740542310703104}]
I20260807 02:24:28.355966 18137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:24:28.356148 18137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:24:28.358510 18137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f5483b891c645aebabee96557c6d391: Sending DeleteTablet for 3 replicas of tablet b5feb08020fe4cfda96a77407f2eb7e8
I20260807 02:24:28.359243 18276 tablet_service.cc:1514] Processing DeleteTablet for tablet b5feb08020fe4cfda96a77407f2eb7e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:24:28 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:53810
I20260807 02:24:28.359257 18554 tablet_service.cc:1514] Processing DeleteTablet for tablet b5feb08020fe4cfda96a77407f2eb7e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:24:28 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:49306
I20260807 02:24:28.359431 18416 tablet_service.cc:1514] Processing DeleteTablet for tablet b5feb08020fe4cfda96a77407f2eb7e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:24:28 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:36288
I20260807 02:24:28.359719 18695 tablet_replica.cc:330] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336: stopping tablet replica
I20260807 02:24:28.359998 18695 raft_consensus.cc:2229] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:24:28.360116 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 18180
I20260807 02:24:28.360327 18695 raft_consensus.cc:2258] T b5feb08020fe4cfda96a77407f2eb7e8 P eb2fef31a22343e3a41fb1d6079b4336 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:24:28.360661 18696 tablet_replica.cc:330] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f: stopping tablet replica
I20260807 02:24:28.360853 18697 tablet_replica.cc:330] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db: stopping tablet replica
I20260807 02:24:28.360947 18696 raft_consensus.cc:2229] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:24:28.361092 18697 raft_consensus.cc:2229] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:24:28.361275 18696 raft_consensus.cc:2258] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:24:28.361322 18697 raft_consensus.cc:2258] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:24:28.362912 18697 ts_tablet_manager.cc:1905] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:24:28.366397 18697 ts_tablet_manager.cc:1918] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:24:28.366499 18697 log.cc:1196] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/b5feb08020fe4cfda96a77407f2eb7e8
I20260807 02:24:28.366655 18696 ts_tablet_manager.cc:1905] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:24:28.366833 18697 ts_tablet_manager.cc:1939] T b5feb08020fe4cfda96a77407f2eb7e8 P ad8a12cafc9744a88568c59431d540db: Deleting consensus metadata
I20260807 02:24:28.368079 18123 catalog_manager.cc:4841] TS ad8a12cafc9744a88568c59431d540db (127.4.221.66:44525): tablet b5feb08020fe4cfda96a77407f2eb7e8 (table test-table [id=4d9886cfa9214dbcbcbd73fd181a8be0]) successfully deleted
I20260807 02:24:28.368956 18696 ts_tablet_manager.cc:1918] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:24:28.369067 18696 log.cc:1196] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/b5feb08020fe4cfda96a77407f2eb7e8
I20260807 02:24:28.369390 18696 ts_tablet_manager.cc:1939] T b5feb08020fe4cfda96a77407f2eb7e8 P 10d2c48788a243cfaec9bc3aea1c156f: Deleting consensus metadata
W20260807 02:24:28.369580 18126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:45017 (error 108)
I20260807 02:24:28.369967 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 18350
I20260807 02:24:28.370908 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:28.370305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:45017 (local address 127.0.0.1:53830)
0807 02:24:28.370705 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:28.370797 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:45017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282}
W20260807 02:24:28.371134 18126 catalog_manager.cc:4584] TS eb2fef31a22343e3a41fb1d6079b4336 (127.4.221.65:45017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5feb08020fe4cfda96a77407f2eb7e8: Network error: Client connection negotiation failed: client connection to 127.4.221.65:45017: connect: Connection refused (error 111)
I20260807 02:24:28.372524 18125 catalog_manager.cc:4841] TS 10d2c48788a243cfaec9bc3aea1c156f (127.4.221.67:41597): tablet b5feb08020fe4cfda96a77407f2eb7e8 (table test-table [id=4d9886cfa9214dbcbcbd73fd181a8be0]) successfully deleted
I20260807 02:24:28.377205 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 18491
I20260807 02:24:28.383564 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 18106
I20260807 02:24:28.391208 4981 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260807 02:24:28.739858 4981 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260807 02:24:28.739984 4981 mini_ranger.cc:66] Stopping Ranger...
I20260807 02:24:29.095798 4981 mini_ranger.cc:68] Stopped Ranger
2026-08-07 02:24:29.096 UTC [17147] LOG: received smart shutdown request
2026-08-07 02:24:29.101 UTC [17147] LOG: background worker "logical replication launcher" (PID 17156) exited with exit code 1
2026-08-07 02:24:29.101 UTC [17151] LOG: shutting down
2026-08-07 02:24:29.290 UTC [17147] LOG: database system is shut down
2026-08-07T02:24:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51188 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:24:29 dist-test-slave-69rg krb5kdc[18708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:24:29 dist-test-slave-69rg krb5kdc[18708](info): set up 2 sockets
Aug 07 02:24:29 dist-test-slave-69rg krb5kdc[18708](info): commencing operation
krb5kdc: starting...
W20260807 02:24:30.532239 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:24:30 dist-test-slave-69rg krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:24:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:24:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:24:30.676983 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:36923
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:36923,127.4.221.125:37421,127.4.221.124:42993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:30.808652 18721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:30.809015 18721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:30.809067 18721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:30.813333 18721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:24:30.813419 18721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:30.813441 18721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:30.813458 18721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:24:30.813510 18721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:24:30.819336 18721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:36923,127.4.221.125:37421,127.4.221.124:42993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:36923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:30.820783 18721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:30.822166 18721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:30.832355 18730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:30.832384 18728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:30.832340 18727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:30.832455 18721 server_base.cc:1034] running on GCE node
I20260807 02:24:30.833389 18721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:30.834856 18721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:30.836050 18721 hybrid_clock.cc:648] HybridClock initialized: now 1786069470836030 us; error 33 us; skew 500 ppm
Aug 07 02:24:30 dist-test-slave-69rg krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069470, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:30.839424 18721 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:24:30.840500 18721 webserver.cc:460] Webserver started at http://127.4.221.126:44565/ using document root <none> and password file <none>
I20260807 02:24:30.841136 18721 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:30.841202 18721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:30.841403 18721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:24:30.843945 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "495171ad0df4402a8238d8946c58c93b"
format_stamp: "Formatted at 2026-08-07 02:24:30 on dist-test-slave-69rg"
server_key: "fc7d3ee7d44a6a57162a9e357e8e0fd3"
server_key_iv: "7e9e6796e411e48cb03b49dc590e2e46"
server_key_version: "encryptionkey@0"
I20260807 02:24:30.844568 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "495171ad0df4402a8238d8946c58c93b"
format_stamp: "Formatted at 2026-08-07 02:24:30 on dist-test-slave-69rg"
server_key: "fc7d3ee7d44a6a57162a9e357e8e0fd3"
server_key_iv: "7e9e6796e411e48cb03b49dc590e2e46"
server_key_version: "encryptionkey@0"
I20260807 02:24:30.849007 18721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260807 02:24:30.852059 18737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:30.853674 18721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:24:30.853881 18721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "495171ad0df4402a8238d8946c58c93b"
format_stamp: "Formatted at 2026-08-07 02:24:30 on dist-test-slave-69rg"
server_key: "fc7d3ee7d44a6a57162a9e357e8e0fd3"
server_key_iv: "7e9e6796e411e48cb03b49dc590e2e46"
server_key_version: "encryptionkey@0"
I20260807 02:24:30.854012 18721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:30.868446 18721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:30.871985 18721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:30.872179 18721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:30.880972 18721 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:36923
I20260807 02:24:30.881006 18791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:36923 every 8 connection(s)
I20260807 02:24:30.882148 18721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:24:30.885694 18792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:30.885931 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 18721
I20260807 02:24:30.886091 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:24:30.886457 4981 external_mini_cluster.cc:1428] Setting key d65714cdfe60407d3c00b41f54a425f9
I20260807 02:24:30.891299 18792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.221.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.125@KRBTEST.COM" created.
Aug 07 02:24:30 dist-test-slave-69rg krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069470, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:30.910032 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:30.894140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.4.221.126:36923)
0807 02:24:30.894722 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:30.894744 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:30.895041 (+ 297us) server_negotiation.cc:407] Connection header received
0807 02:24:30.896209 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:30.896215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:30.896498 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:30.896741 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:30.897701 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:30.898775 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:30.899889 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:30.900472 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:30.900601 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:30.904079 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:30.904089 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:30.904107 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:30.904140 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:30.906328 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:30.906841 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:30.906851 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:30.906856 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:30.907057 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:30.907397 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:30.907401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:30.907405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:30.908061 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:30.908230 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:30.908626 (+ 396us) server_negotiation.cc:299] Negotiation successful
0807 02:24:30.908991 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":161,"server-negotiator.queue_time_us":374,"thread_start_us":153,"threads_started":1}
I20260807 02:24:30.910032 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:30.894140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36923 (local address 127.0.0.1:56722)
0807 02:24:30.894845 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:30.894898 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:30.895853 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:30.896790 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:30.896797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:30.897153 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:30.897531 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:30.897546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:30.898897 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:30.898900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:30.899765 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:30.899775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:30.900653 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:30.900660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:30.900824 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:30.901567 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:30.901595 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:30.903888 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:30.906462 (+ 2574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:30.906471 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:30.906480 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:30.906724 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:30.907171 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:30.907174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:30.907175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:30.907294 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:30.908199 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:30.908205 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:30.908343 (+ 138us) client_negotiation.cc:769] Sending connection context
0807 02:24:30.908514 (+ 171us) client_negotiation.cc:240] Negotiation successful
0807 02:24:30.908884 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":98,"spinlock_wait_cycles":37120,"thread_start_us":204,"threads_started":1}
I20260807 02:24:30.914188 18792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:24:30.915297 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:30.914954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:37421 (local address 127.0.0.1:42648)
0807 02:24:30.915098 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:30.915197 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 02:24:30.915639 18740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.221.125:37421: connect: Connection refused (error 111)
I20260807 02:24:30.916829 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:30.916570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:37421 (local address 127.0.0.1:42656)
0807 02:24:30.916698 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:30.916762 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260807 02:24:30.917057 18792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.125:37421: Network error: Client connection negotiation failed: client connection to 127.4.221.125:37421: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.125.keytab.
Entry for principal kudu/127.4.221.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.125.keytab.
I20260807 02:24:30.921715 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.125:37421
--webserver_interface=127.4.221.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.125.keytab
--principal=kudu/127.4.221.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:36923,127.4.221.125:37421,127.4.221.124:42993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:24:30.966388 18792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } attempt: 1
I20260807 02:24:30.967630 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:30.967186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:37421 (local address 127.0.0.1:42666)
0807 02:24:30.967405 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:30.967502 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 02:24:30.968591 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:30.968332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:37421 (local address 127.0.0.1:42678)
0807 02:24:30.968453 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:30.968515 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 02:24:30.968751 18792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.125:37421: Network error: Client connection negotiation failed: client connection to 127.4.221.125:37421: connect: Connection refused (error 111)
I20260807 02:24:31.036971 18792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } attempt: 2
I20260807 02:24:31.038142 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.037679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:37421 (local address 127.0.0.1:42688)
0807 02:24:31.037924 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.038007 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260807 02:24:31.039031 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.038770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:37421 (local address 127.0.0.1:42700)
0807 02:24:31.038892 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.038956 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 02:24:31.039167 18792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.125:37421: Network error: Client connection negotiation failed: client connection to 127.4.221.125:37421: connect: Connection refused (error 111)
W20260807 02:24:31.056489 18799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:31.056808 18799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:31.056854 18799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:31.061400 18799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:24:31.061473 18799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:31.061493 18799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:31.061510 18799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:24:31.061554 18799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:24:31.067360 18799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:36923,127.4.221.125:37421,127.4.221.124:42993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.125.keytab
--principal=kudu/127.4.221.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.125:37421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.221.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:31.068627 18799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:31.069802 18799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:31.079066 18805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:31.079099 18808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:31.079090 18804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:31.079427 18799 server_base.cc:1034] running on GCE node
I20260807 02:24:31.080430 18799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:31.081710 18799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:31.082929 18799 hybrid_clock.cc:648] HybridClock initialized: now 1786069471082912 us; error 39 us; skew 500 ppm
Aug 07 02:24:31 dist-test-slave-69rg krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069471, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:31.086048 18799 init.cc:377] Logged in from keytab as kudu/127.4.221.125@KRBTEST.COM (short username kudu)
I20260807 02:24:31.087064 18799 webserver.cc:460] Webserver started at http://127.4.221.125:36009/ using document root <none> and password file <none>
I20260807 02:24:31.087702 18799 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:31.087797 18799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:31.088053 18799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:24:31.090453 18799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/data/instance:
uuid: "8c18897a8b6c43e0a8a0006b9612b485"
format_stamp: "Formatted at 2026-08-07 02:24:31 on dist-test-slave-69rg"
server_key: "9fb810d02eb9b3a21f5e6862dd666590"
server_key_iv: "8bfd585ecfbe45d66c7af030b8c9684f"
server_key_version: "encryptionkey@0"
I20260807 02:24:31.091070 18799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/wal/instance:
uuid: "8c18897a8b6c43e0a8a0006b9612b485"
format_stamp: "Formatted at 2026-08-07 02:24:31 on dist-test-slave-69rg"
server_key: "9fb810d02eb9b3a21f5e6862dd666590"
server_key_iv: "8bfd585ecfbe45d66c7af030b8c9684f"
server_key_version: "encryptionkey@0"
I20260807 02:24:31.095181 18799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:24:31.097992 18815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:31.099396 18799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 02:24:31.099551 18799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/wal
uuid: "8c18897a8b6c43e0a8a0006b9612b485"
format_stamp: "Formatted at 2026-08-07 02:24:31 on dist-test-slave-69rg"
server_key: "9fb810d02eb9b3a21f5e6862dd666590"
server_key_iv: "8bfd585ecfbe45d66c7af030b8c9684f"
server_key_version: "encryptionkey@0"
I20260807 02:24:31.099674 18799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:31.124158 18799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:31.127820 18799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:31.128070 18799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:31.130380 18792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } attempt: 3
I20260807 02:24:31.131206 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.130906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:37421 (local address 127.0.0.1:42712)
0807 02:24:31.131063 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.131127 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 02:24:31.132091 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.131850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:37421 (local address 127.0.0.1:42726)
0807 02:24:31.131966 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.132023 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 02:24:31.132296 18792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.125:37421: Network error: Client connection negotiation failed: client connection to 127.4.221.125:37421: connect: Connection refused (error 111)
I20260807 02:24:31.136232 18799 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.125:37421
I20260807 02:24:31.136251 18869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.125:37421 every 8 connection(s)
I20260807 02:24:31.137276 18799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/data/info.pb
I20260807 02:24:31.140664 18870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:31.141184 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 18799
I20260807 02:24:31.141340 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-1/wal/instance
I20260807 02:24:31.141656 4981 external_mini_cluster.cc:1428] Setting key b5923afa0493998835744248f74c4fba
I20260807 02:24:31.145892 18870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.221.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.124@KRBTEST.COM" created.
Aug 07 02:24:31 dist-test-slave-69rg krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069471, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.125@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:31.163527 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.148747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.4.221.126:36923)
0807 02:24:31.148957 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:31.148965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:31.149922 (+ 957us) server_negotiation.cc:407] Connection header received
0807 02:24:31.150993 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:31.150997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:31.151066 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:31.151189 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:31.152509 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.153174 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.154472 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.154852 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.154985 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.158000 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:31.158004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:31.158006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:31.158040 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:31.160849 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.161455 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.161459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.161461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.161540 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.161948 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.161952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.161956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.162220 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:31.162336 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:31.163228 (+ 892us) server_negotiation.cc:299] Negotiation successful
0807 02:24:31.163390 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 02:24:31.164764 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.149141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36923 (local address 127.0.0.1:56736)
0807 02:24:31.149711 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.149780 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:31.150758 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:31.151366 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:31.151373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:31.151755 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:31.152356 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.152371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.153311 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.153314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.154346 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.154357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.155030 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.155035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.155225 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.155905 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:31.155923 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:31.157740 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:31.160992 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.160999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.161016 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.161325 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.161642 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.161647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.161649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.161806 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.162372 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:31.162381 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:31.162801 (+ 420us) client_negotiation.cc:769] Sending connection context
0807 02:24:31.163050 (+ 249us) client_negotiation.cc:240] Negotiation successful
0807 02:24:31.163464 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":173,"threads_started":1}
I20260807 02:24:31.166632 18870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:24:31 dist-test-slave-69rg krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069471, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.125@KRBTEST.COM for kudu/127.4.221.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.124.keytab.
Entry for principal kudu/127.4.221.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.124.keytab.
I20260807 02:24:31.173797 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.124:42993
--webserver_interface=127.4.221.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.124.keytab
--principal=kudu/127.4.221.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:35131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:36923,127.4.221.125:37421,127.4.221.124:42993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:24:31.176656 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.167308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:37421 (local address 127.0.0.1:42732)
0807 02:24:31.167416 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.167432 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:31.167510 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:31.168170 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:31.168172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:31.168352 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:31.168435 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.168440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.169527 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.169530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.170214 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.170224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.170805 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.170808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.170906 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.171460 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:31.171472 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:31.172771 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:31.174948 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.174954 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.174959 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.175230 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.175600 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.175605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.175610 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.175694 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.176271 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:31.176273 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:31.176350 (+ 77us) client_negotiation.cc:769] Sending connection context
0807 02:24:31.176430 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 02:24:31.176536 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260807 02:24:31.176967 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.167475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.4.221.125:37421)
0807 02:24:31.167836 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:31.167846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:31.167864 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:24:31.167897 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:31.167901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:31.168055 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:31.168202 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:31.168541 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.169367 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.170335 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.170706 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.170788 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.172918 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:31.172931 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:31.172950 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:31.172982 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:31.174833 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.175366 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.175371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.175372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.175473 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.175786 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.175791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.175793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.176149 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:31.176309 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:31.176561 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 02:24:31.176736 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":139,"threads_started":1}
I20260807 02:24:31.179019 18870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:24:31.179766 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.179513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:42993 (local address 127.0.0.1:45912)
0807 02:24:31.179636 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.179712 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 02:24:31.179948 18816 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
I20260807 02:24:31.180958 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.180710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:42993 (local address 127.0.0.1:45920)
0807 02:24:31.180841 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.180905 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260807 02:24:31.181134 18870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:42993: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
I20260807 02:24:31.230429 18870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } attempt: 1
I20260807 02:24:31.231287 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.230981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:42993 (local address 127.0.0.1:45924)
0807 02:24:31.231143 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.231210 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260807 02:24:31.232015 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.231803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:42993 (local address 127.0.0.1:45934)
0807 02:24:31.231911 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.231961 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 02:24:31.232129 18870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:42993: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
I20260807 02:24:31.275489 18792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } attempt: 4
Aug 07 02:24:31 dist-test-slave-69rg krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069470, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for kudu/127.4.221.125@KRBTEST.COM
I20260807 02:24:31.285434 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.276108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:37421 (local address 127.0.0.1:42744)
0807 02:24:31.276309 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.276327 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:31.276438 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:31.276709 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:31.276711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:31.276888 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:31.277011 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.277016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.278562 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.278566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.279236 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.279247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.279904 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.279907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.279994 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.280558 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:31.280571 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:31.282076 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:31.283919 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.283921 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.283923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.284114 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.284401 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.284403 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.284404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.284490 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.284954 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:31.284957 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:31.285056 (+ 99us) client_negotiation.cc:769] Sending connection context
0807 02:24:31.285168 (+ 112us) client_negotiation.cc:240] Negotiation successful
0807 02:24:31.285292 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260807 02:24:31.285512 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.276186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.4.221.125:37421)
0807 02:24:31.276328 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:31.276333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:31.276391 (+ 58us) server_negotiation.cc:407] Connection header received
0807 02:24:31.276538 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:31.276543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:31.276601 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:31.276711 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:31.277133 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.278432 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.279366 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.279769 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.279873 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.282214 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:31.282218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:31.282221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:31.282248 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:31.283806 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.284213 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.284217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.284219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.284308 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.284579 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.284584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.284587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.284820 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:31.284932 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:31.285244 (+ 312us) server_negotiation.cc:299] Negotiation successful
0807 02:24:31.285364 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 02:24:31.286365 18792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:24:31.287072 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.286835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:42993 (local address 127.0.0.1:45944)
0807 02:24:31.286959 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.287016 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 02:24:31.287806 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.287574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:42993 (local address 127.0.0.1:45956)
0807 02:24:31.287700 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.287744 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260807 02:24:31.287949 18792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:42993: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
W20260807 02:24:31.297106 18876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:31.297365 18876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:31.297407 18876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 02:24:31.300329 18870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } attempt: 2
I20260807 02:24:31.301155 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.300873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:42993 (local address 127.0.0.1:45968)
0807 02:24:31.301013 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.301073 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 02:24:31.301908 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.301624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:42993 (local address 127.0.0.1:45976)
0807 02:24:31.301750 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.301802 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 02:24:31.301923 18876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:24:31.301994 18876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:31.302027 18876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:31.302037 18870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:42993: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
W20260807 02:24:31.302059 18876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:24:31.302091 18876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:24:31.307595 18876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:35131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:36923,127.4.221.125:37421,127.4.221.124:42993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.124.keytab
--principal=kudu/127.4.221.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.124:42993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.221.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:31.308624 18876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:31.309607 18876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:31.317684 18885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:31.317690 18882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:31.317741 18883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:31.318001 18876 server_base.cc:1034] running on GCE node
I20260807 02:24:31.318562 18876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:31.319607 18876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:31.320788 18876 hybrid_clock.cc:648] HybridClock initialized: now 1786069471320758 us; error 41 us; skew 500 ppm
Aug 07 02:24:31 dist-test-slave-69rg krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069471, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:31.323760 18876 init.cc:377] Logged in from keytab as kudu/127.4.221.124@KRBTEST.COM (short username kudu)
I20260807 02:24:31.324643 18876 webserver.cc:460] Webserver started at http://127.4.221.124:37717/ using document root <none> and password file <none>
I20260807 02:24:31.325198 18876 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:31.325258 18876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:31.325495 18876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:24:31.327594 18876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/data/instance:
uuid: "a56802c4a4ae4115a6c75317dbd17ff9"
format_stamp: "Formatted at 2026-08-07 02:24:31 on dist-test-slave-69rg"
server_key: "76d7496ace2e2ac330782a1285c54aef"
server_key_iv: "ad1f9decaa9a66f4b9ac7b56b87c6fca"
server_key_version: "encryptionkey@0"
I20260807 02:24:31.328176 18876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/wal/instance:
uuid: "a56802c4a4ae4115a6c75317dbd17ff9"
format_stamp: "Formatted at 2026-08-07 02:24:31 on dist-test-slave-69rg"
server_key: "76d7496ace2e2ac330782a1285c54aef"
server_key_iv: "ad1f9decaa9a66f4b9ac7b56b87c6fca"
server_key_version: "encryptionkey@0"
I20260807 02:24:31.332001 18876 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:24:31.334682 18892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:31.336001 18876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:24:31.336167 18876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/wal
uuid: "a56802c4a4ae4115a6c75317dbd17ff9"
format_stamp: "Formatted at 2026-08-07 02:24:31 on dist-test-slave-69rg"
server_key: "76d7496ace2e2ac330782a1285c54aef"
server_key_iv: "ad1f9decaa9a66f4b9ac7b56b87c6fca"
server_key_version: "encryptionkey@0"
I20260807 02:24:31.336292 18876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:31.347129 18792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } attempt: 1
I20260807 02:24:31.348173 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.347762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:42993 (local address 127.0.0.1:45986)
0807 02:24:31.347996 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.348061 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260807 02:24:31.349054 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.348760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:42993 (local address 127.0.0.1:45992)
0807 02:24:31.348935 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.348986 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260807 02:24:31.349244 18792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:42993: Network error: Client connection negotiation failed: client connection to 127.4.221.124:42993: connect: Connection refused (error 111)
I20260807 02:24:31.351440 18876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:31.354563 18876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:31.354724 18876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:31.361703 18876 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.124:42993
I20260807 02:24:31.361742 18946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.124:42993 every 8 connection(s)
I20260807 02:24:31.362677 18876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/data/info.pb
I20260807 02:24:31.365257 18947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:31.368866 18947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:24:31.372066 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 18876
I20260807 02:24:31.372206 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786069301823597-4981-0/minicluster-data/master-2/wal/instance
I20260807 02:24:31.372493 4981 external_mini_cluster.cc:1428] Setting key 5cfd6340e40400e91a520038afef60c5
Aug 07 02:24:31 dist-test-slave-69rg krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069471, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.124@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
Aug 07 02:24:31 dist-test-slave-69rg krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.124@KRBTEST.COM
I20260807 02:24:31.382344 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.370620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56748 (local address 127.4.221.126:36923)
0807 02:24:31.370748 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:31.370752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:31.371508 (+ 756us) server_negotiation.cc:407] Connection header received
0807 02:24:31.372712 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:31.372715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:31.372762 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:31.372833 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:31.374178 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.374697 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.375745 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.375969 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.376083 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.378760 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:31.378763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:31.378765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:31.378789 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:31.380244 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.380773 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.380777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.380778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.380863 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.381225 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.381228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.381230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.381421 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:31.381530 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:31.382043 (+ 513us) server_negotiation.cc:299] Negotiation successful
0807 02:24:31.382176 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 02:24:31.383239 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.370922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:36923 (local address 127.0.0.1:56748)
0807 02:24:31.371365 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.371406 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:31.372485 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:31.373013 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:31.373019 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:31.373501 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:31.374031 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.374046 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.374827 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.374837 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.375615 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.375624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.376116 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.376120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.376292 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.376844 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:31.376859 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:31.378610 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:31.380358 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.380369 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.380378 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.380663 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.380956 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.380960 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.380962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.381096 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.381530 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:31.381536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:31.381763 (+ 227us) client_negotiation.cc:769] Sending connection context
0807 02:24:31.381983 (+ 220us) client_negotiation.cc:240] Negotiation successful
0807 02:24:31.382265 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":112,"threads_started":1}
I20260807 02:24:31.383262 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.373245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.4.221.124:42993)
0807 02:24:31.373617 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:31.373629 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:31.373668 (+ 39us) server_negotiation.cc:407] Connection header received
0807 02:24:31.373713 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:31.373719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:31.373955 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:31.374128 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:31.374843 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.375631 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.376449 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.376949 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.377070 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.379805 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:31.379810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:31.379818 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:31.379838 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:31.381384 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.381841 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.381850 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.381853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.381939 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.382221 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.382225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.382227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.382517 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:31.382706 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:31.382906 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 02:24:31.383010 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":152,"threads_started":1}
I20260807 02:24:31.384434 18947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:24:31 dist-test-slave-69rg krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069471, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.124@KRBTEST.COM for kudu/127.4.221.125@KRBTEST.COM
Aug 07 02:24:31 dist-test-slave-69rg krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:31.393180 18870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } attempt: 3
I20260807 02:24:31.393805 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.385139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42758 (local address 127.4.221.125:37421)
0807 02:24:31.385311 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:31.385317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:31.385334 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:24:31.385467 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:31.385470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:31.385525 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:31.385609 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:31.386082 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.386709 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.387781 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.388000 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.388124 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.390237 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:31.390241 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:31.390244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:31.390274 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:31.391905 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.392379 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.392382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.392383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.392456 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.392789 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.392792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.392794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.393027 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:31.393132 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:31.393527 (+ 395us) server_negotiation.cc:299] Negotiation successful
0807 02:24:31.393642 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 02:24:31.393792 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.384938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:37421 (local address 127.0.0.1:42758)
0807 02:24:31.385280 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.385294 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:31.385361 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:31.385646 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:31.385649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:31.385823 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:31.385957 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.385965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.386851 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.386854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.387654 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.387665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.388146 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.388149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.388227 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.388673 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:31.388691 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:31.390092 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:31.392014 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.392020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.392025 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.392274 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.392549 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.392554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.392557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.392665 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.393218 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:31.393221 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:31.393316 (+ 95us) client_negotiation.cc:769] Sending connection context
0807 02:24:31.393463 (+ 147us) client_negotiation.cc:240] Negotiation successful
0807 02:24:31.393609 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275}
I20260807 02:24:31.394505 18947 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:24:31.395516 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.385372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56762 (local address 127.4.221.126:36923)
0807 02:24:31.385482 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:31.385486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:31.385529 (+ 43us) server_negotiation.cc:407] Connection header received
0807 02:24:31.385655 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:31.385658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:31.385701 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:31.385775 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:31.386443 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.387049 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.388064 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.388338 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.388466 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.390853 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:31.390857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:31.390859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:31.390889 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:31.392679 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.393234 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.393237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.393239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.393328 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.394118 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.394122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.394123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.394362 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:31.394547 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:31.395253 (+ 706us) server_negotiation.cc:299] Negotiation successful
0807 02:24:31.395394 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Aug 07 02:24:31 dist-test-slave-69rg krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069471, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.125@KRBTEST.COM for kudu/127.4.221.124@KRBTEST.COM
Aug 07 02:24:31 dist-test-slave-69rg krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069471, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.124@KRBTEST.COM for kudu/127.4.221.124@KRBTEST.COM
Aug 07 02:24:31 dist-test-slave-69rg krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.125@KRBTEST.COM
I20260807 02:24:31.403853 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.393676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.4.221.124:42993)
0807 02:24:31.394219 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:31.394224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:31.394235 (+ 11us) server_negotiation.cc:407] Connection header received
0807 02:24:31.394266 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:31.394268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:31.394304 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:31.394386 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:31.394908 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.395544 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.397135 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.397564 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.398074 (+ 510us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.399971 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:31.399974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:31.399977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:31.400008 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:31.401944 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.402416 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.402420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.402422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.402506 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.402995 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.403000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.403002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.403234 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:31.403328 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:31.403598 (+ 270us) server_negotiation.cc:299] Negotiation successful
0807 02:24:31.403702 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":477}
I20260807 02:24:31.403854 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.393589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:42993 (local address 127.0.0.1:46018)
0807 02:24:31.393758 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.393772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:31.393878 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:31.394526 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:31.394531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:31.394690 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:31.394787 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.394794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.396201 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.396205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.396975 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.396988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.397704 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.397708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.397851 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.398318 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:31.398341 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:31.399819 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:31.402075 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.402080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.402084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.402309 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.402600 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.402604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.402607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.402694 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.403320 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:31.403322 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:31.403419 (+ 97us) client_negotiation.cc:769] Sending connection context
0807 02:24:31.403541 (+ 122us) client_negotiation.cc:240] Negotiation successful
0807 02:24:31.403695 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"spinlock_wait_cycles":2432}
I20260807 02:24:31.405179 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.394940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:42993 (local address 127.0.0.1:46034)
0807 02:24:31.395726 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.395746 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:31.395811 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:31.396031 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:31.396033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:31.396151 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:31.396283 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.396290 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.397556 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.397562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.398325 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.398335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.398896 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.398899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.398994 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.399400 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:31.399410 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:31.400688 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:31.402806 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.402812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.402816 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.403058 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.403372 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.403375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.403376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.403455 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.404295 (+ 840us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:31.404298 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:31.404389 (+ 91us) client_negotiation.cc:769] Sending connection context
0807 02:24:31.404496 (+ 107us) client_negotiation.cc:240] Negotiation successful
0807 02:24:31.404620 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":3}
I20260807 02:24:31.407342 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.396858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42768 (local address 127.4.221.125:37421)
0807 02:24:31.396962 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:31.396965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:31.396977 (+ 12us) server_negotiation.cc:407] Connection header received
0807 02:24:31.397011 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:31.397020 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:31.397056 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:31.397132 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:31.397876 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.398595 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.399921 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.400284 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.400396 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.403396 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:31.403402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:31.403407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:31.403436 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:31.405664 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.406134 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.406136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.406137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.406238 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.406566 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.406569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.406570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.406738 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:31.406905 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:31.407097 (+ 192us) server_negotiation.cc:299] Negotiation successful
0807 02:24:31.407192 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 02:24:31.407573 18870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485: Bootstrap starting.
I20260807 02:24:31.410027 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.395209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46034 (local address 127.4.221.124:42993)
0807 02:24:31.395548 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:31.395557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:31.395809 (+ 252us) server_negotiation.cc:407] Connection header received
0807 02:24:31.395871 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:31.395874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:31.395920 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:31.396013 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:31.396398 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.397456 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.398440 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.398789 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.398896 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.400819 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:31.400822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:31.400824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:31.400846 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:31.402687 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.403166 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.403170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.403194 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.403277 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.404004 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.404008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.404009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.404208 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:31.404302 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:31.404572 (+ 270us) server_negotiation.cc:299] Negotiation successful
0807 02:24:31.404695 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":279,"thread_start_us":131,"threads_started":1}
I20260807 02:24:31.410411 18870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:31.410785 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:24:31.411296 18870 log.cc:824] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:31.413129 18947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9: Bootstrap starting.
I20260807 02:24:31.413446 18870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485: No bootstrap required, opened a new log
I20260807 02:24:31.415925 18947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:31.416414 18792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } attempt: 2
I20260807 02:24:31.416757 18947 log.cc:824] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:31.420416 18870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } }
I20260807 02:24:31.420665 18870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:31.420719 18870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c18897a8b6c43e0a8a0006b9612b485, State: Initialized, Role: FOLLOWER
I20260807 02:24:31.421868 18870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } }
I20260807 02:24:31.423305 18870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485 [sys.catalog]: configured and running, proceeding with master startup.
Aug 07 02:24:31 dist-test-slave-69rg krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069470, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for kudu/127.4.221.124@KRBTEST.COM
I20260807 02:24:31.425741 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.413504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42776 (local address 127.4.221.125:37421)
0807 02:24:31.413603 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:31.413608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:31.414395 (+ 787us) server_negotiation.cc:407] Connection header received
0807 02:24:31.414484 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:31.414489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:31.414543 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:31.414618 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:31.415599 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.416355 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.417501 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.417928 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.418022 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.420029 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:31.420033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:31.420035 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:31.420060 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:31.422696 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.423738 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.423743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.423746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.423830 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.424165 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.424168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.424169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.424394 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:31.424512 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:31.424900 (+ 388us) server_negotiation.cc:299] Negotiation successful
0807 02:24:31.425292 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260807 02:24:31.426122 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.413944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.4.221.126:36923)
0807 02:24:31.417208 (+ 3264us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:31.417212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:31.417226 (+ 14us) server_negotiation.cc:407] Connection header received
0807 02:24:31.417261 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:31.417264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:31.417304 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:31.417379 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:31.418350 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.419125 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.420294 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.420670 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.420757 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.422523 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:31.422529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:31.422535 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:31.422564 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:31.424207 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.424669 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.424672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.424674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.424760 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.425070 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.425075 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.425076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.425307 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:31.425427 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:31.425798 (+ 371us) server_negotiation.cc:299] Negotiation successful
0807 02:24:31.425969 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3177}
I20260807 02:24:31.426972 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.414101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46036 (local address 127.4.221.124:42993)
0807 02:24:31.414230 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:31.414234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:31.414675 (+ 441us) server_negotiation.cc:407] Connection header received
0807 02:24:31.414823 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:31.414826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:31.414872 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:31.414919 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:31.415838 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.416569 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.417887 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.418246 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.418318 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.421008 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:31.421010 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:31.421013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:31.421036 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:31.424183 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.425106 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.425110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.425111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.425200 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.425549 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.425553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.425555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.425786 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:31.425910 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:31.426693 (+ 783us) server_negotiation.cc:299] Negotiation successful
0807 02:24:31.426806 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 02:24:31.428154 18796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.416793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:42993 (local address 127.0.0.1:46038)
0807 02:24:31.417041 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:31.417054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:31.417134 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:31.417567 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:31.417570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:31.417768 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:31.417940 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.417947 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.418909 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.418913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.419736 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:31.419747 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.421487 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.421491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:31.421577 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.422272 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:31.422286 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:31.424584 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:31.426542 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.426545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.426546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.426806 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.427100 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:31.427105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:31.427109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:31.427211 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:31.427659 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:31.427665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:31.427764 (+ 99us) client_negotiation.cc:769] Sending connection context
0807 02:24:31.427882 (+ 118us) client_negotiation.cc:240] Negotiation successful
0807 02:24:31.428009 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260807 02:24:31.428171 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:31.417266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.4.221.124:42993)
0807 02:24:31.417367 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:31.417371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:31.417382 (+ 11us) server_negotiation.cc:407] Connection header received
0807 02:24:31.417413 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:31.417415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:31.417449 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:31.418036 (+ 587us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:31.418129 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.418790 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.421007 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:31.421355 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:31.421473 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:31.424707 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:31.424711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:31.424713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:31.424746 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:31.426421 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.426913 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.426916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.426918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.426992 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:31.427298 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:31.427301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:31.427303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:31.427508 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:31.427608 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:31.427937 (+ 329us) server_negotiation.cc:299] Negotiation successful
0807 02:24:31.428040 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":42}
I20260807 02:24:31.430864 18947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9: No bootstrap required, opened a new log
I20260807 02:24:31.431384 18792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b: Bootstrap starting.
W20260807 02:24:31.432955 18976 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:24:31.433024 18976 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:24:31.429436 18799 master_runner.cc:186] Error getting master registration for 127.4.221.126:36923: OK, instance_id { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" instance_seqno: 1786069470878532 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 02:24:31.433480 18799 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 02:24:31.433645 18964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } } }
I20260807 02:24:31.433755 18964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:24:31.434181 18792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:31.434809 18792 log.cc:824] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:31.435047 18947 raft_consensus.cc:348] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } }
I20260807 02:24:31.435241 18947 raft_consensus.cc:374] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:31.435276 18947 raft_consensus.cc:729] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a56802c4a4ae4115a6c75317dbd17ff9, State: Initialized, Role: FOLLOWER
I20260807 02:24:31.435737 18947 consensus_queue.cc:260] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } }
I20260807 02:24:31.436830 18792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b: No bootstrap required, opened a new log
I20260807 02:24:31.436794 18980 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } } }
I20260807 02:24:31.436925 18980 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:24:31.437037 18947 sys_catalog.cc:564] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:24:31.440790 18792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } }
I20260807 02:24:31.441035 18792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:31.441107 18792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495171ad0df4402a8238d8946c58c93b, State: Initialized, Role: FOLLOWER
I20260807 02:24:31.441576 18792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } }
I20260807 02:24:31.442844 18792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [sys.catalog]: configured and running, proceeding with master startup.
W20260807 02:24:31.443274 18991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:24:31.443388 18991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:24:31.440475 18876 master_runner.cc:186] Error getting master registration for 127.4.221.126:36923: OK, instance_id { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" instance_seqno: 1786069470878532 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 02:24:31.443784 18876 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 02:24:31.445253 18993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } } }
I20260807 02:24:31.445389 18993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [sys.catalog]: This master's current role is: FOLLOWER
W20260807 02:24:31.447827 19004 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:24:31.447952 19004 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:24:31.451726 18721 master_runner.cc:263] No leader master found from master 495171ad0df4402a8238d8946c58c93b
I20260807 02:24:31.451797 18721 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260807 02:24:31.606001 18898 tablet.cc:2369] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:24:31.624583 18743 tablet.cc:2369] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:24:31.630048 18822 tablet.cc:2369] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:24:31.640208 18993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:24:31.640596 18993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } }
I20260807 02:24:31.641711 18993 leader_election.cc:288] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c18897a8b6c43e0a8a0006b9612b485 (127.4.221.125:37421), a56802c4a4ae4115a6c75317dbd17ff9 (127.4.221.124:42993)
I20260807 02:24:31.642534 18843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "495171ad0df4402a8238d8946c58c93b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c18897a8b6c43e0a8a0006b9612b485" is_pre_election: true
I20260807 02:24:31.642773 18919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "495171ad0df4402a8238d8946c58c93b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" is_pre_election: true
I20260807 02:24:31.642899 18843 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 495171ad0df4402a8238d8946c58c93b in term 0.
I20260807 02:24:31.643076 18919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 495171ad0df4402a8238d8946c58c93b in term 0.
I20260807 02:24:31.643635 18740 leader_election.cc:302] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 495171ad0df4402a8238d8946c58c93b, 8c18897a8b6c43e0a8a0006b9612b485; no voters:
I20260807 02:24:31.643955 18993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:24:31.644044 18993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:24:31.644093 18993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:31.645428 18993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } }
I20260807 02:24:31.645809 18993 leader_election.cc:288] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [CANDIDATE]: Term 1 election: Requested vote from peers 8c18897a8b6c43e0a8a0006b9612b485 (127.4.221.125:37421), a56802c4a4ae4115a6c75317dbd17ff9 (127.4.221.124:42993)
I20260807 02:24:31.646178 18843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "495171ad0df4402a8238d8946c58c93b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c18897a8b6c43e0a8a0006b9612b485"
I20260807 02:24:31.646299 18843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:31.646745 18919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "495171ad0df4402a8238d8946c58c93b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a56802c4a4ae4115a6c75317dbd17ff9"
I20260807 02:24:31.646881 18919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:31.647614 18843 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 495171ad0df4402a8238d8946c58c93b in term 1.
I20260807 02:24:31.648162 18919 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 495171ad0df4402a8238d8946c58c93b in term 1.
I20260807 02:24:31.648536 18740 leader_election.cc:302] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 495171ad0df4402a8238d8946c58c93b, 8c18897a8b6c43e0a8a0006b9612b485; no voters:
I20260807 02:24:31.648734 18993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:24:31.649006 18993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [term 1 LEADER]: Becoming Leader. State: Replica: 495171ad0df4402a8238d8946c58c93b, State: Running, Role: LEADER
I20260807 02:24:31.649346 18993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } }
I20260807 02:24:31.652597 19006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 495171ad0df4402a8238d8946c58c93b. Latest consensus state: current_term: 1 leader_uuid: "495171ad0df4402a8238d8946c58c93b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } } }
I20260807 02:24:31.652740 19006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:31.653153 19008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:24:31.654433 19008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:24:31.659153 18919 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9 [term 1 FOLLOWER]: Refusing update from remote peer 495171ad0df4402a8238d8946c58c93b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:24:31.659452 18843 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485 [term 1 FOLLOWER]: Refusing update from remote peer 495171ad0df4402a8238d8946c58c93b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:24:31.659997 18993 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:24:31.660220 19006 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:24:31.663995 18964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 495171ad0df4402a8238d8946c58c93b. Latest consensus state: current_term: 1 leader_uuid: "495171ad0df4402a8238d8946c58c93b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } } }
I20260807 02:24:31.664127 18964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:24:31.664624 18980 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 495171ad0df4402a8238d8946c58c93b. Latest consensus state: current_term: 1 leader_uuid: "495171ad0df4402a8238d8946c58c93b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } } }
I20260807 02:24:31.664749 18980 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:24:31.669783 18993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "495171ad0df4402a8238d8946c58c93b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } } }
I20260807 02:24:31.669975 18993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:31.670025 19009 mvcc.cc:205] Tried to move back new op lower bound from 7315740555909152768 to 7315740555878572032. Current Snapshot: MvccSnapshot[applied={T|T < 7315740555909152768}]
I20260807 02:24:31.670297 18993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "495171ad0df4402a8238d8946c58c93b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } } }
I20260807 02:24:31.670418 18993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:31.671597 19010 mvcc.cc:205] Tried to move back new op lower bound from 7315740555909152768 to 7315740555878572032. Current Snapshot: MvccSnapshot[applied={T|T < 7315740555909152768}]
I20260807 02:24:31.671617 18964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "495171ad0df4402a8238d8946c58c93b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } } }
I20260807 02:24:31.671731 18964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:24:31.671754 18980 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "495171ad0df4402a8238d8946c58c93b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495171ad0df4402a8238d8946c58c93b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 36923 } } peers { permanent_uuid: "8c18897a8b6c43e0a8a0006b9612b485" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 37421 } } peers { permanent_uuid: "a56802c4a4ae4115a6c75317dbd17ff9" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 42993 } } }
I20260807 02:24:31.671873 18980 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:24:31.672128 19008 catalog_manager.cc:1347] Generated new cluster ID: 993cc761a125405e9c6e9abe87f2b676
I20260807 02:24:31.672194 19008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:24:31.672331 19011 mvcc.cc:205] Tried to move back new op lower bound from 7315740555909152768 to 7315740555878572032. Current Snapshot: MvccSnapshot[applied={T|T < 7315740555909152768}]
I20260807 02:24:31.689878 19008 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:24:31.691133 19008 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:24:31.701947 19008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 495171ad0df4402a8238d8946c58c93b: Generated new TSK 0
I20260807 02:24:31.702848 19008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:24:32.434788 18976 catalog_manager.cc:1259] Loaded cluster ID: 993cc761a125405e9c6e9abe87f2b676
I20260807 02:24:32.434890 18976 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485: loading cluster ID for follower catalog manager: success
I20260807 02:24:32.436740 18976 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485: acquiring CA information for follower catalog manager: success
I20260807 02:24:32.438326 18976 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8c18897a8b6c43e0a8a0006b9612b485: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 02:24:32.445104 18991 catalog_manager.cc:1259] Loaded cluster ID: 993cc761a125405e9c6e9abe87f2b676
I20260807 02:24:32.445187 18991 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9: loading cluster ID for follower catalog manager: success
I20260807 02:24:32.447078 18991 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9: acquiring CA information for follower catalog manager: success
I20260807 02:24:32.448531 18991 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a56802c4a4ae4115a6c75317dbd17ff9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 02:24:32.773136 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 18721
I20260807 02:24:32.782586 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 18799
I20260807 02:24:32.790777 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 18876
2026-08-07T02:24:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3356 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:24:32 dist-test-slave-69rg krb5kdc[19023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:24:32 dist-test-slave-69rg krb5kdc[19023](info): set up 2 sockets
Aug 07 02:24:32 dist-test-slave-69rg krb5kdc[19023](info): commencing operation
krb5kdc: starting...
W20260807 02:24:33.859688 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:24:33 dist-test-slave-69rg krb5kdc[19023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:24:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:24:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:24:33.990260 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:40299
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:40299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:34.105037 19036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:34.105340 19036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:34.105422 19036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:34.109470 19036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:24:34.109562 19036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:34.109601 19036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:34.109658 19036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:24:34.109697 19036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:24:34.115001 19036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:40299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:40299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:34.116182 19036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:34.117208 19036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:34.125728 19042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:34.125682 19045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:34.125733 19043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:34.126116 19036 server_base.cc:1034] running on GCE node
I20260807 02:24:34.126674 19036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:34.127667 19036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:34.128851 19036 hybrid_clock.cc:648] HybridClock initialized: now 1786069474128816 us; error 42 us; skew 500 ppm
Aug 07 02:24:34 dist-test-slave-69rg krb5kdc[19023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069474, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:34.131891 19036 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:24:34.132701 19036 webserver.cc:460] Webserver started at http://127.4.221.126:43883/ using document root <none> and password file <none>
I20260807 02:24:34.133212 19036 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:34.133273 19036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:34.133435 19036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:24:34.135454 19036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "7e889c24415345a18c166d8d629b5e7e"
format_stamp: "Formatted at 2026-08-07 02:24:34 on dist-test-slave-69rg"
server_key: "38a63bc143cc62cf81062d1b9e379705"
server_key_iv: "30e654729b0142a315d062fe15111ba2"
server_key_version: "encryptionkey@0"
I20260807 02:24:34.136052 19036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "7e889c24415345a18c166d8d629b5e7e"
format_stamp: "Formatted at 2026-08-07 02:24:34 on dist-test-slave-69rg"
server_key: "38a63bc143cc62cf81062d1b9e379705"
server_key_iv: "30e654729b0142a315d062fe15111ba2"
server_key_version: "encryptionkey@0"
I20260807 02:24:34.139693 19036 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260807 02:24:34.142052 19052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:34.143260 19036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:24:34.143397 19036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "7e889c24415345a18c166d8d629b5e7e"
format_stamp: "Formatted at 2026-08-07 02:24:34 on dist-test-slave-69rg"
server_key: "38a63bc143cc62cf81062d1b9e379705"
server_key_iv: "30e654729b0142a315d062fe15111ba2"
server_key_version: "encryptionkey@0"
I20260807 02:24:34.143505 19036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:34.154536 19036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:34.157895 19036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:34.158123 19036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:34.165124 19036 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:40299
I20260807 02:24:34.165200 19107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:40299 every 8 connection(s)
I20260807 02:24:34.166158 19036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:24:34.166358 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 19036
I20260807 02:24:34.166486 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:24:34.166846 4981 external_mini_cluster.cc:1428] Setting key 128c11eb69e648e5ab2c0731b41dbd2f
I20260807 02:24:34.169180 19108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 02:24:34 dist-test-slave-69rg krb5kdc[19023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:34.175799 19108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e: Bootstrap starting.
I20260807 02:24:34.178227 19108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:34.178951 19108 log.cc:824] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:34.180805 19108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e: No bootstrap required, opened a new log
I20260807 02:24:34.181131 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:34.168134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.4.221.126:40299)
0807 02:24:34.168553 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:34.168568 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:34.168602 (+ 34us) server_negotiation.cc:407] Connection header received
0807 02:24:34.169227 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:34.169243 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:34.169608 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:34.169965 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:34.170863 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:34.171835 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:34.173026 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:34.173479 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:34.173595 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:34.176447 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:34.176452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:34.176464 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:34.176490 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:34.178280 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:34.178816 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:34.178821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:34.178826 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:34.178935 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:34.179239 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:34.179242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:34.179244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:34.179621 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:34.179822 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:34.180131 (+ 309us) server_negotiation.cc:299] Negotiation successful
0807 02:24:34.180419 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":147,"threads_started":1}
I20260807 02:24:34.184046 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:24:34.184980 19108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e889c24415345a18c166d8d629b5e7e" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40299 } }
I20260807 02:24:34.185262 19108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:34.185334 19108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e889c24415345a18c166d8d629b5e7e, State: Initialized, Role: FOLLOWER
I20260807 02:24:34.185772 19108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e889c24415345a18c166d8d629b5e7e" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40299 } }
I20260807 02:24:34.185967 19108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:24:34.186019 19108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:24:34.186092 19108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:34.186918 19108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e889c24415345a18c166d8d629b5e7e" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40299 } }
I20260807 02:24:34.187212 19108 leader_election.cc:302] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e889c24415345a18c166d8d629b5e7e; no voters:
I20260807 02:24:34.187479 19108 leader_election.cc:288] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:24:34.187606 19113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:24:34.187891 19113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [term 1 LEADER]: Becoming Leader. State: Replica: 7e889c24415345a18c166d8d629b5e7e, State: Running, Role: LEADER
I20260807 02:24:34.188274 19113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e889c24415345a18c166d8d629b5e7e" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40299 } }
I20260807 02:24:34.188669 19108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:24:34.189950 19115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e889c24415345a18c166d8d629b5e7e. Latest consensus state: current_term: 1 leader_uuid: "7e889c24415345a18c166d8d629b5e7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e889c24415345a18c166d8d629b5e7e" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40299 } } }
I20260807 02:24:34.190080 19115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:34.190415 19114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e889c24415345a18c166d8d629b5e7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e889c24415345a18c166d8d629b5e7e" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40299 } } }
I20260807 02:24:34.190483 19124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:24:34.190515 19114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:34.193045 19124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:24:34.198194 19124 catalog_manager.cc:1347] Generated new cluster ID: 144e1d44226f4055932f460fffdedf8c
I20260807 02:24:34.198262 19124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:24:34.209561 19124 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:24:34.210834 19124 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:24:34.220400 19124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e889c24415345a18c166d8d629b5e7e: Generated new TSK 0
I20260807 02:24:34.221179 19124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:24:34.240088 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 19036
2026-08-07T02:24:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1445 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:24:34 dist-test-slave-69rg krb5kdc[19138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:24:34 dist-test-slave-69rg krb5kdc[19138](info): set up 2 sockets
Aug 07 02:24:34 dist-test-slave-69rg krb5kdc[19138](info): commencing operation
krb5kdc: starting...
W20260807 02:24:35.313043 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:24:35 dist-test-slave-69rg krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:24:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:24:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:24:35.453202 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44123
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:44123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:35.569186 19151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:35.569551 19151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:35.569631 19151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:35.573799 19151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:24:35.573947 19151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:35.573987 19151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:35.574025 19151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:24:35.574060 19151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:24:35.579651 19151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:44123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:35.580859 19151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:35.581995 19151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:35.590812 19157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:35.590812 19159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:35.590808 19156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:35.591204 19151 server_base.cc:1034] running on GCE node
I20260807 02:24:35.591910 19151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:35.593111 19151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:35.594333 19151 hybrid_clock.cc:648] HybridClock initialized: now 1786069475594305 us; error 44 us; skew 500 ppm
Aug 07 02:24:35 dist-test-slave-69rg krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069475, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:35.597236 19151 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:24:35.598246 19151 webserver.cc:460] Webserver started at http://127.4.221.126:37975/ using document root <none> and password file <none>
I20260807 02:24:35.598829 19151 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:35.598923 19151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:35.599190 19151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:24:35.601446 19151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "4b369e37b60c4060b96afbf56d850544"
format_stamp: "Formatted at 2026-08-07 02:24:35 on dist-test-slave-69rg"
server_key: "8af10ef8c4415e0fad84388990f613c0"
server_key_iv: "51ef8c0a8b574de1e47d79f7f4640137"
server_key_version: "encryptionkey@0"
I20260807 02:24:35.602108 19151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "4b369e37b60c4060b96afbf56d850544"
format_stamp: "Formatted at 2026-08-07 02:24:35 on dist-test-slave-69rg"
server_key: "8af10ef8c4415e0fad84388990f613c0"
server_key_iv: "51ef8c0a8b574de1e47d79f7f4640137"
server_key_version: "encryptionkey@0"
I20260807 02:24:35.606004 19151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 02:24:35.608598 19166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:35.609961 19151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:24:35.610109 19151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "4b369e37b60c4060b96afbf56d850544"
format_stamp: "Formatted at 2026-08-07 02:24:35 on dist-test-slave-69rg"
server_key: "8af10ef8c4415e0fad84388990f613c0"
server_key_iv: "51ef8c0a8b574de1e47d79f7f4640137"
server_key_version: "encryptionkey@0"
I20260807 02:24:35.610239 19151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:35.632531 19151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:35.636260 19151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:35.636442 19151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:35.644415 19151 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:44123
I20260807 02:24:35.644745 19219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:44123 every 8 connection(s)
I20260807 02:24:35.645581 19151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:24:35.648456 19220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:35.651353 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 19151
I20260807 02:24:35.651466 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:24:35.651782 4981 external_mini_cluster.cc:1428] Setting key a0db24d2ee6b742587ae12a3badc39ea
I20260807 02:24:35.654865 19220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: Bootstrap starting.
I20260807 02:24:35.657666 19220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:35.658526 19220 log.cc:824] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:35.660969 19220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: No bootstrap required, opened a new log
Aug 07 02:24:35 dist-test-slave-69rg krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:35.664928 19220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } }
I20260807 02:24:35.665228 19220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:35.665328 19220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b369e37b60c4060b96afbf56d850544, State: Initialized, Role: FOLLOWER
I20260807 02:24:35.665863 19220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } }
I20260807 02:24:35.666045 19220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:24:35.666159 19220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:24:35.666296 19220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:35.667491 19220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } }
I20260807 02:24:35.667886 19220 leader_election.cc:302] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b369e37b60c4060b96afbf56d850544; no voters:
I20260807 02:24:35.668202 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:35.653276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.4.221.126:44123)
0807 02:24:35.653739 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:35.653761 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:35.653805 (+ 44us) server_negotiation.cc:407] Connection header received
0807 02:24:35.654644 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:35.654667 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:35.655096 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:35.655468 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:35.656192 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:35.657701 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:35.658657 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:35.659295 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:35.659496 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:35.662264 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:35.662269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:35.662283 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:35.662311 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:35.664740 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:35.665232 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:35.665240 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:35.665255 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:35.665410 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:35.665745 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:35.665748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:35.665750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:35.666407 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:35.666651 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:35.666990 (+ 339us) server_negotiation.cc:299] Negotiation successful
0807 02:24:35.667304 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":129,"threads_started":1}
I20260807 02:24:35.668287 19220 leader_election.cc:288] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:24:35.668386 19226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:24:35.668644 19226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 1 LEADER]: Becoming Leader. State: Replica: 4b369e37b60c4060b96afbf56d850544, State: Running, Role: LEADER
I20260807 02:24:35.669021 19226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } }
I20260807 02:24:35.669770 19220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:24:35.671001 19226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b369e37b60c4060b96afbf56d850544. Latest consensus state: current_term: 1 leader_uuid: "4b369e37b60c4060b96afbf56d850544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } } }
I20260807 02:24:35.671134 19226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:35.671790 19228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b369e37b60c4060b96afbf56d850544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } } }
I20260807 02:24:35.672155 19228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:35.672215 19241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:24:35.675199 19241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:24:35.680936 19241 catalog_manager.cc:1347] Generated new cluster ID: 63b8648bcf624c6abf5e0ce4d3458459
I20260807 02:24:35.681223 19241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:24:35.696934 19241 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:24:35.698166 19241 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:24:35.704980 19241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: Generated new TSK 0
I20260807 02:24:35.705888 19241 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:24:35.746083 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44123
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:35.863292 19250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:35.863584 19250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:35.863674 19250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:35.867844 19250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:35.867919 19250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:35.868012 19250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:24:35.873981 19250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:35.875175 19250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:35.876245 19250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:35.886310 19255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:35.886313 19258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:35.886353 19256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:35.886564 19250 server_base.cc:1034] running on GCE node
I20260807 02:24:35.887064 19250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:35.887784 19250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:35.888991 19250 hybrid_clock.cc:648] HybridClock initialized: now 1786069475888955 us; error 55 us; skew 500 ppm
Aug 07 02:24:35 dist-test-slave-69rg krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069475, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:35.892055 19250 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:24:35.892999 19250 webserver.cc:460] Webserver started at http://127.4.221.65:45795/ using document root <none> and password file <none>
I20260807 02:24:35.893625 19250 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:35.893698 19250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:35.893983 19250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:24:35.896070 19250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "197797ffdacf4f3cb08d28ad7b071103"
format_stamp: "Formatted at 2026-08-07 02:24:35 on dist-test-slave-69rg"
server_key: "7f167da237ac21a3a4fed6b424367627"
server_key_iv: "e47c55ef4712599d955ec02fcb703078"
server_key_version: "encryptionkey@0"
I20260807 02:24:35.896677 19250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "197797ffdacf4f3cb08d28ad7b071103"
format_stamp: "Formatted at 2026-08-07 02:24:35 on dist-test-slave-69rg"
server_key: "7f167da237ac21a3a4fed6b424367627"
server_key_iv: "e47c55ef4712599d955ec02fcb703078"
server_key_version: "encryptionkey@0"
I20260807 02:24:35.900518 19250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260807 02:24:35.903137 19266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:35.904451 19250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 02:24:35.904590 19250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "197797ffdacf4f3cb08d28ad7b071103"
format_stamp: "Formatted at 2026-08-07 02:24:35 on dist-test-slave-69rg"
server_key: "7f167da237ac21a3a4fed6b424367627"
server_key_iv: "e47c55ef4712599d955ec02fcb703078"
server_key_version: "encryptionkey@0"
I20260807 02:24:35.904724 19250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:35.916352 19250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:35.919991 19250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:35.920228 19250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:35.920892 19250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:35.921810 19250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:24:35.921939 19250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:35.922014 19250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:24:35.922055 19250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:35.932368 19250 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:38365
I20260807 02:24:35.932406 19387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:38365 every 8 connection(s)
I20260807 02:24:35.933334 19250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:24:35.934466 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 19250
I20260807 02:24:35.934605 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:24:35.934955 4981 external_mini_cluster.cc:1428] Setting key 553c57881d860b898ed4fc9e0e1c5c0d
Aug 07 02:24:35 dist-test-slave-69rg krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069475, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
I20260807 02:24:35.949069 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:35.935546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:37351 (local address 127.4.221.126:44123)
0807 02:24:35.935699 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:35.935704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:35.937116 (+ 1412us) server_negotiation.cc:407] Connection header received
0807 02:24:35.938513 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:35.938516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:35.938563 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:35.938639 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:35.940014 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:35.940576 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:35.941970 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:35.942391 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:35.942526 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:35.945167 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:35.945172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:35.945175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:35.945212 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:35.946864 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:35.947416 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:35.947421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:35.947426 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:35.947556 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:35.947902 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:35.947904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:35.947905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:35.948089 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:35.948169 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:35.948751 (+ 582us) server_negotiation.cc:299] Negotiation successful
0807 02:24:35.948920 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 02:24:35.950045 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:35.935991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:37351)
0807 02:24:35.936944 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:35.937003 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:35.938289 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:35.938822 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:35.938833 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:35.939348 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:35.939856 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:35.939871 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:35.940726 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:35.940730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:35.941809 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:35.941821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:35.942657 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:35.942665 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:35.942832 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:35.943463 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:35.943480 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:35.944974 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:35.946975 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:35.946979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:35.946991 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:35.947318 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:35.947649 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:35.947651 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:35.947653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:35.947806 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:35.948183 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:35.948188 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:35.948526 (+ 338us) client_negotiation.cc:769] Sending connection context
0807 02:24:35.948690 (+ 164us) client_negotiation.cc:240] Negotiation successful
0807 02:24:35.949052 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":160,"threads_started":1}
I20260807 02:24:35.951398 19388 heartbeater.cc:347] Connected to a master server at 127.4.221.126:44123
I20260807 02:24:35.951691 19388 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:35.952342 19388 heartbeater.cc:511] Master 127.4.221.126:44123 requested a full tablet report, sending...
I20260807 02:24:35.954113 19183 ts_manager.cc:194] Registered new tserver with Master: 197797ffdacf4f3cb08d28ad7b071103 (127.4.221.65:38365)
I20260807 02:24:35.955890 19183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:37351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:24:35.969784 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44123
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:36.092582 19393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:36.092922 19393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:36.093005 19393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:36.097261 19393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:36.097362 19393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:36.097461 19393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:24:36.103153 19393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:36.104377 19393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:36.105468 19393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:36.116096 19399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:36.116109 19401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:36.116187 19398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:36.116330 19393 server_base.cc:1034] running on GCE node
I20260807 02:24:36.116977 19393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:36.117640 19393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:36.118814 19393 hybrid_clock.cc:648] HybridClock initialized: now 1786069476118801 us; error 32 us; skew 500 ppm
Aug 07 02:24:36 dist-test-slave-69rg krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:36.121764 19393 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:24:36.122776 19393 webserver.cc:460] Webserver started at http://127.4.221.66:32851/ using document root <none> and password file <none>
I20260807 02:24:36.123363 19393 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:36.123427 19393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:36.123677 19393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:24:36.125994 19393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "631ebf3e199c415abdc52720f90bc90e"
format_stamp: "Formatted at 2026-08-07 02:24:36 on dist-test-slave-69rg"
server_key: "bfd3023565641e90f6fc7bcd21e548c4"
server_key_iv: "4d1c086f9c457fb89870afdb2a0113dd"
server_key_version: "encryptionkey@0"
I20260807 02:24:36.126632 19393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "631ebf3e199c415abdc52720f90bc90e"
format_stamp: "Formatted at 2026-08-07 02:24:36 on dist-test-slave-69rg"
server_key: "bfd3023565641e90f6fc7bcd21e548c4"
server_key_iv: "4d1c086f9c457fb89870afdb2a0113dd"
server_key_version: "encryptionkey@0"
I20260807 02:24:36.130755 19393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 02:24:36.133409 19408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:36.134743 19393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260807 02:24:36.134856 19393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "631ebf3e199c415abdc52720f90bc90e"
format_stamp: "Formatted at 2026-08-07 02:24:36 on dist-test-slave-69rg"
server_key: "bfd3023565641e90f6fc7bcd21e548c4"
server_key_iv: "4d1c086f9c457fb89870afdb2a0113dd"
server_key_version: "encryptionkey@0"
I20260807 02:24:36.134976 19393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:36.155265 19393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:36.159422 19393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:36.159734 19393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:36.160646 19393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:36.161798 19393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:24:36.161949 19393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:36.162034 19393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:24:36.162086 19393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:36.175505 19393 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:33005
I20260807 02:24:36.175531 19525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:33005 every 8 connection(s)
I20260807 02:24:36.176745 19393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:24:36.178275 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 19393
I20260807 02:24:36.178407 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:24:36.178779 4981 external_mini_cluster.cc:1428] Setting key 95f9281f4f4e34badcd651e70bcf62ee
Aug 07 02:24:36 dist-test-slave-69rg krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
I20260807 02:24:36.194937 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.179443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:56787 (local address 127.4.221.126:44123)
0807 02:24:36.179621 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:36.179626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:36.181053 (+ 1427us) server_negotiation.cc:407] Connection header received
0807 02:24:36.182142 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:36.182145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:36.182209 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:36.182292 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:36.183633 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.184300 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.186026 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.186297 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.186416 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.190046 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:36.190051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:36.190054 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:36.190084 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:36.192051 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:36.192766 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:36.192777 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:36.192779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:36.192881 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:36.193316 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:36.193321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:36.193323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:36.193655 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:36.193805 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:36.194585 (+ 780us) server_negotiation.cc:299] Negotiation successful
0807 02:24:36.194777 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 02:24:36.196419 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.180002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:56787)
0807 02:24:36.180891 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:36.180937 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:36.181926 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:36.182480 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:36.182488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:36.182914 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:36.183473 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:36.183488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.184427 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.184431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:36.185870 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:36.185886 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.186497 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.186506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:36.186792 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.187620 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:36.187647 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:36.189733 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:36.192203 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:36.192212 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:36.192231 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:36.192631 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:36.192995 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:36.193001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:36.193004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:36.193182 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:36.193799 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:36.193808 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:36.194241 (+ 433us) client_negotiation.cc:769] Sending connection context
0807 02:24:36.194486 (+ 245us) client_negotiation.cc:240] Negotiation successful
0807 02:24:36.194939 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":182,"threads_started":1}
I20260807 02:24:36.198161 19526 heartbeater.cc:347] Connected to a master server at 127.4.221.126:44123
I20260807 02:24:36.198513 19526 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:36.199283 19526 heartbeater.cc:511] Master 127.4.221.126:44123 requested a full tablet report, sending...
I20260807 02:24:36.200922 19183 ts_manager.cc:194] Registered new tserver with Master: 631ebf3e199c415abdc52720f90bc90e (127.4.221.66:33005)
I20260807 02:24:36.201962 19183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:56787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:24:36.213905 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44123
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:36.344491 19531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:36.344892 19531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:36.344977 19531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:36.349246 19531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:36.349371 19531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:36.349478 19531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:24:36.355355 19531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:36.356858 19531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:36.358235 19531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:36.370270 19537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:36.370289 19536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:36.371199 19539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:36.372362 19531 server_base.cc:1034] running on GCE node
I20260807 02:24:36.372892 19531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:36.373785 19531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:36.375066 19531 hybrid_clock.cc:648] HybridClock initialized: now 1786069476375052 us; error 36 us; skew 500 ppm
Aug 07 02:24:36 dist-test-slave-69rg krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:36.378422 19531 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:24:36.379513 19531 webserver.cc:460] Webserver started at http://127.4.221.67:44153/ using document root <none> and password file <none>
I20260807 02:24:36.380167 19531 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:36.380228 19531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:36.380483 19531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:24:36.383018 19531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "f34d7c5265af4d91b341795bcf5f9f1a"
format_stamp: "Formatted at 2026-08-07 02:24:36 on dist-test-slave-69rg"
server_key: "85a138e2ed3a42ba1f058677ccbd56f8"
server_key_iv: "bb9635437cf0e524c607be9f9cd9c66e"
server_key_version: "encryptionkey@0"
I20260807 02:24:36.383653 19531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "f34d7c5265af4d91b341795bcf5f9f1a"
format_stamp: "Formatted at 2026-08-07 02:24:36 on dist-test-slave-69rg"
server_key: "85a138e2ed3a42ba1f058677ccbd56f8"
server_key_iv: "bb9635437cf0e524c607be9f9cd9c66e"
server_key_version: "encryptionkey@0"
I20260807 02:24:36.388396 19531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:24:36.391557 19547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:36.393262 19531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260807 02:24:36.393393 19531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "f34d7c5265af4d91b341795bcf5f9f1a"
format_stamp: "Formatted at 2026-08-07 02:24:36 on dist-test-slave-69rg"
server_key: "85a138e2ed3a42ba1f058677ccbd56f8"
server_key_iv: "bb9635437cf0e524c607be9f9cd9c66e"
server_key_version: "encryptionkey@0"
I20260807 02:24:36.393512 19531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:36.407629 19531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:36.411906 19531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:36.412178 19531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:36.412968 19531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:36.414089 19531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:24:36.414170 19531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:36.414247 19531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:24:36.414296 19531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:36.425038 19531 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:41487
I20260807 02:24:36.425062 19673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:41487 every 8 connection(s)
I20260807 02:24:36.426232 19531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:24:36.433650 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 19531
I20260807 02:24:36.433789 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:24:36.434167 4981 external_mini_cluster.cc:1428] Setting key af8b12c8c7106890352fac5de6977cd2
Aug 07 02:24:36 dist-test-slave-69rg krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069476, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:36.443552 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.428790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:59137 (local address 127.4.221.126:44123)
0807 02:24:36.428983 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:36.428987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:36.430123 (+ 1136us) server_negotiation.cc:407] Connection header received
0807 02:24:36.431615 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:36.431619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:36.431694 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:36.431783 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:36.433317 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.434224 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.435810 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.436127 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.436241 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.439247 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:36.439250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:36.439297 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:36.439334 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:36.441168 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:36.441741 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:36.441747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:36.441749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:36.441888 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:36.442269 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:36.442272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:36.442274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:36.442499 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:36.442615 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:36.443194 (+ 579us) server_negotiation.cc:299] Negotiation successful
0807 02:24:36.443349 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 02:24:36.444613 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.429225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:59137)
0807 02:24:36.429935 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:36.429988 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:36.431357 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:36.432017 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:36.432025 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:36.432477 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:36.433072 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:36.433089 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.434430 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.434433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:36.435670 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:36.435681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.436289 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.436292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:36.436485 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.437271 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:36.437290 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:36.439048 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:36.441297 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:36.441310 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:36.441331 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:36.441627 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:36.441991 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:36.441995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:36.441997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:36.442165 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:36.442612 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:36.442619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:36.442930 (+ 311us) client_negotiation.cc:769] Sending connection context
0807 02:24:36.443151 (+ 221us) client_negotiation.cc:240] Negotiation successful
0807 02:24:36.443537 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":198,"threads_started":1}
I20260807 02:24:36.446026 19674 heartbeater.cc:347] Connected to a master server at 127.4.221.126:44123
I20260807 02:24:36.446346 19674 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:36.447018 19674 heartbeater.cc:511] Master 127.4.221.126:44123 requested a full tablet report, sending...
I20260807 02:24:36.448369 19183 ts_manager.cc:194] Registered new tserver with Master: f34d7c5265af4d91b341795bcf5f9f1a (127.4.221.67:41487)
I20260807 02:24:36.448963 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:24:36.449079 19183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:59137
I20260807 02:24:36.459585 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.451732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37556 (local address 127.4.221.126:44123)
0807 02:24:36.451863 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:36.451867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:36.452087 (+ 220us) server_negotiation.cc:407] Connection header received
0807 02:24:36.452193 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:36.452195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:36.452237 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:36.452302 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:36.453025 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.453577 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.454622 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.454854 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.454973 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.456512 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:36.456515 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:36.456518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:36.456540 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:36.458028 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:36.458472 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:36.458475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:36.458477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:36.458553 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:36.458784 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:36.458786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:36.458787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:36.459032 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:36.459143 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:36.459343 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 02:24:36.459447 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 02:24:36.463009 19183 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:24:36.466749 19183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:24:36.484035 19687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.477174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:38365 (local address 127.0.0.1:39178)
0807 02:24:36.478144 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:36.478189 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:36.478353 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:36.479623 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:36.479626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:36.479643 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:36.479764 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:36.479771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.481352 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.481358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:36.482299 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:36.482308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.483395 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.483398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:36.483513 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.483571 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 02:24:36.483662 (+ 91us) client_negotiation.cc:240] Negotiation successful
0807 02:24:36.483834 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":91,"spinlock_wait_cycles":8832,"thread_start_us":145,"threads_started":1}
I20260807 02:24:36.484182 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.478343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:33005 (local address 127.0.0.1:39278)
0807 02:24:36.478788 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:36.478798 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:36.478870 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:36.479592 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:36.479595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:36.479614 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:36.479719 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:36.479729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.481218 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.481221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:36.482559 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:36.482570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.483652 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.483656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:36.483772 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.483800 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 02:24:36.483887 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 02:24:36.483963 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":73,"thread_start_us":119,"threads_started":1}
I20260807 02:24:36.484529 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.478784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.4.221.66:33005)
0807 02:24:36.479145 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:36.479159 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:36.479180 (+ 21us) server_negotiation.cc:407] Connection header received
0807 02:24:36.479263 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:36.479267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:36.479453 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:36.479609 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:36.479816 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.481079 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.482689 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.483521 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.483626 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.483799 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:36.483919 (+ 120us) server_negotiation.cc:299] Negotiation successful
0807 02:24:36.484019 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":136,"threads_started":1}
I20260807 02:24:36.484807 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.477129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:41487 (local address 127.0.0.1:57806)
0807 02:24:36.478129 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:36.478187 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:36.478353 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:36.479274 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:36.479279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:36.479299 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:36.479418 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:36.479423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.481017 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.481023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:36.482559 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:36.482570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.484277 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.484280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:36.484401 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.484419 (+ 18us) client_negotiation.cc:769] Sending connection context
0807 02:24:36.484559 (+ 140us) client_negotiation.cc:240] Negotiation successful
0807 02:24:36.484660 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":7,"spinlock_wait_cycles":1792,"thread_start_us":85,"threads_started":1}
I20260807 02:24:36.484884 19688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.477367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57806 (local address 127.4.221.67:41487)
0807 02:24:36.477836 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:36.477851 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:36.478520 (+ 669us) server_negotiation.cc:407] Connection header received
0807 02:24:36.478613 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:36.478618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:36.478869 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:36.479064 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:36.479909 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.480880 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.482728 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.483955 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.484062 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.484212 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:36.484598 (+ 386us) server_negotiation.cc:299] Negotiation successful
0807 02:24:36.484714 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":220,"threads_started":1}
I20260807 02:24:36.485592 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.478106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39178 (local address 127.4.221.65:38365)
0807 02:24:36.479135 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:36.479145 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:36.479174 (+ 29us) server_negotiation.cc:407] Connection header received
0807 02:24:36.479263 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:36.479268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:36.479453 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:36.479928 (+ 475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:36.480120 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.481233 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.482432 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.483242 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.485139 (+ 1897us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.485292 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:36.485341 (+ 49us) server_negotiation.cc:299] Negotiation successful
0807 02:24:36.485439 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":899,"thread_start_us":133,"threads_started":1}
I20260807 02:24:36.488162 19316 tablet_service.cc:1467] Processing CreateTablet for tablet d95bd793855e48aab516a5c990e1fa02 (DEFAULT_TABLE table=test-table [id=f771321e85764048baa9c6da3a2a428e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:24:36.488168 19458 tablet_service.cc:1467] Processing CreateTablet for tablet d95bd793855e48aab516a5c990e1fa02 (DEFAULT_TABLE table=test-table [id=f771321e85764048baa9c6da3a2a428e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:24:36.488169 19601 tablet_service.cc:1467] Processing CreateTablet for tablet d95bd793855e48aab516a5c990e1fa02 (DEFAULT_TABLE table=test-table [id=f771321e85764048baa9c6da3a2a428e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:24:36.489349 19316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d95bd793855e48aab516a5c990e1fa02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:36.489351 19458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d95bd793855e48aab516a5c990e1fa02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:36.489359 19601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d95bd793855e48aab516a5c990e1fa02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:36.496472 19692 tablet_bootstrap.cc:492] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: Bootstrap starting.
I20260807 02:24:36.497805 19693 tablet_bootstrap.cc:492] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e: Bootstrap starting.
I20260807 02:24:36.497960 19694 tablet_bootstrap.cc:492] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a: Bootstrap starting.
I20260807 02:24:36.499984 19692 tablet_bootstrap.cc:654] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:36.500731 19694 tablet_bootstrap.cc:654] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:36.500747 19693 tablet_bootstrap.cc:654] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:36.500890 19692 log.cc:824] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:36.501642 19694 log.cc:824] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:36.501642 19693 log.cc:824] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:36.503317 19692 tablet_bootstrap.cc:492] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: No bootstrap required, opened a new log
I20260807 02:24:36.503512 19693 tablet_bootstrap.cc:492] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e: No bootstrap required, opened a new log
I20260807 02:24:36.503556 19692 ts_tablet_manager.cc:1397] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260807 02:24:36.503659 19694 tablet_bootstrap.cc:492] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a: No bootstrap required, opened a new log
I20260807 02:24:36.503737 19693 ts_tablet_manager.cc:1397] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 02:24:36.503840 19694 ts_tablet_manager.cc:1397] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 02:24:36.507257 19693 raft_consensus.cc:348] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } }
I20260807 02:24:36.507566 19693 raft_consensus.cc:374] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:36.507625 19693 raft_consensus.cc:729] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 631ebf3e199c415abdc52720f90bc90e, State: Initialized, Role: FOLLOWER
I20260807 02:24:36.508126 19693 consensus_queue.cc:260] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } }
I20260807 02:24:36.508234 19694 raft_consensus.cc:348] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } }
I20260807 02:24:36.508548 19694 raft_consensus.cc:374] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:36.508617 19694 raft_consensus.cc:729] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f34d7c5265af4d91b341795bcf5f9f1a, State: Initialized, Role: FOLLOWER
I20260807 02:24:36.509104 19526 heartbeater.cc:503] Master 127.4.221.126:44123 was elected leader, sending a full tablet report...
I20260807 02:24:36.509136 19694 consensus_queue.cc:260] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } }
I20260807 02:24:36.509048 19692 raft_consensus.cc:348] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } }
I20260807 02:24:36.509280 19692 raft_consensus.cc:374] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:36.509327 19692 raft_consensus.cc:729] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197797ffdacf4f3cb08d28ad7b071103, State: Initialized, Role: FOLLOWER
I20260807 02:24:36.509382 19693 ts_tablet_manager.cc:1428] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 02:24:36.509862 19674 heartbeater.cc:503] Master 127.4.221.126:44123 was elected leader, sending a full tablet report...
I20260807 02:24:36.509845 19692 consensus_queue.cc:260] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } }
I20260807 02:24:36.510181 19694 ts_tablet_manager.cc:1428] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 02:24:36.510818 19692 ts_tablet_manager.cc:1428] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260807 02:24:36.511209 19388 heartbeater.cc:503] Master 127.4.221.126:44123 was elected leader, sending a full tablet report...
W20260807 02:24:36.677503 19675 tablet.cc:2369] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:24:36.678311 19527 tablet.cc:2369] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:24:36.684898 19389 tablet.cc:2369] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:24:36.776942 19699 raft_consensus.cc:482] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:24:36.777276 19699 raft_consensus.cc:504] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } }
I20260807 02:24:36.778923 19699 leader_election.cc:288] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 197797ffdacf4f3cb08d28ad7b071103 (127.4.221.65:38365), 631ebf3e199c415abdc52720f90bc90e (127.4.221.66:33005)
I20260807 02:24:36.783613 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.779544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:41985 (local address 127.4.221.65:38365)
0807 02:24:36.779738 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:36.779745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:36.779769 (+ 24us) server_negotiation.cc:407] Connection header received
0807 02:24:36.779891 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:36.779897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:36.779977 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:36.780078 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:36.780533 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.781145 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.782414 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.782913 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.783066 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.783143 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:36.783328 (+ 185us) server_negotiation.cc:299] Negotiation successful
0807 02:24:36.783431 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260807 02:24:36.783583 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.779385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:38365 (local address 127.4.221.67:41985)
0807 02:24:36.779581 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:36.779605 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:36.779756 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:36.780195 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:36.780201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:36.780228 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:36.780361 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:36.780367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.781270 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.781276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:36.782267 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:36.782277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.783048 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.783052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:36.783176 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.783208 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 02:24:36.783307 (+ 99us) client_negotiation.cc:240] Negotiation successful
0807 02:24:36.783396 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260807 02:24:36.783614 19701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.779727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:33005 (local address 127.4.221.67:36609)
0807 02:24:36.780211 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:36.780229 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:36.780352 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:36.780650 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:36.780653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:36.780669 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:36.780822 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:36.780827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.781594 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.781599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:36.782463 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:36.782473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.783152 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.783155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:36.783266 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.783291 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 02:24:36.783373 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 02:24:36.783459 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":215,"threads_started":1}
I20260807 02:24:36.783689 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.779942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:36609 (local address 127.4.221.66:33005)
0807 02:24:36.780126 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:36.780131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:36.780319 (+ 188us) server_negotiation.cc:407] Connection header received
0807 02:24:36.780485 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:36.780490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:36.780546 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:36.780639 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:36.780948 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.781473 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.782598 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.783021 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.783158 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.783219 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:36.783403 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 02:24:36.783479 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260807 02:24:36.784510 19478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95bd793855e48aab516a5c990e1fa02" candidate_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "631ebf3e199c415abdc52720f90bc90e" is_pre_election: true
I20260807 02:24:36.784513 19338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95bd793855e48aab516a5c990e1fa02" candidate_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197797ffdacf4f3cb08d28ad7b071103" is_pre_election: true
I20260807 02:24:36.784924 19338 raft_consensus.cc:2454] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f34d7c5265af4d91b341795bcf5f9f1a in term 0.
I20260807 02:24:36.784924 19478 raft_consensus.cc:2454] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f34d7c5265af4d91b341795bcf5f9f1a in term 0.
I20260807 02:24:36.785588 19551 leader_election.cc:302] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197797ffdacf4f3cb08d28ad7b071103, f34d7c5265af4d91b341795bcf5f9f1a; no voters:
I20260807 02:24:36.786016 19699 raft_consensus.cc:2781] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:24:36.786103 19699 raft_consensus.cc:482] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:24:36.786134 19699 raft_consensus.cc:3037] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:36.787520 19699 raft_consensus.cc:504] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } }
I20260807 02:24:36.787940 19699 leader_election.cc:288] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [CANDIDATE]: Term 1 election: Requested vote from peers 197797ffdacf4f3cb08d28ad7b071103 (127.4.221.65:38365), 631ebf3e199c415abdc52720f90bc90e (127.4.221.66:33005)
I20260807 02:24:36.788532 19338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95bd793855e48aab516a5c990e1fa02" candidate_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197797ffdacf4f3cb08d28ad7b071103"
I20260807 02:24:36.788530 19478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d95bd793855e48aab516a5c990e1fa02" candidate_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "631ebf3e199c415abdc52720f90bc90e"
I20260807 02:24:36.788703 19338 raft_consensus.cc:3037] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:36.788709 19478 raft_consensus.cc:3037] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:36.790071 19478 raft_consensus.cc:2454] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f34d7c5265af4d91b341795bcf5f9f1a in term 1.
I20260807 02:24:36.790074 19338 raft_consensus.cc:2454] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f34d7c5265af4d91b341795bcf5f9f1a in term 1.
I20260807 02:24:36.790417 19551 leader_election.cc:302] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 631ebf3e199c415abdc52720f90bc90e, f34d7c5265af4d91b341795bcf5f9f1a; no voters:
I20260807 02:24:36.790606 19699 raft_consensus.cc:2781] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:24:36.790880 19699 raft_consensus.cc:686] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [term 1 LEADER]: Becoming Leader. State: Replica: f34d7c5265af4d91b341795bcf5f9f1a, State: Running, Role: LEADER
I20260807 02:24:36.791292 19699 consensus_queue.cc:237] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } }
I20260807 02:24:36.795630 19180 catalog_manager.cc:5495] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a reported cstate change: term changed from 0 to 1, leader changed from <none> to f34d7c5265af4d91b341795bcf5f9f1a (127.4.221.67). New cstate: current_term: 1 leader_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } health_report { overall_health: UNKNOWN } } }
I20260807 02:24:36.851650 19688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.847323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57812 (local address 127.4.221.67:41487)
0807 02:24:36.847523 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:36.847528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:36.847545 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:24:36.847649 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:36.847652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:36.847718 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:36.847826 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:24:36.848070 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.848658 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.849635 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:36.849986 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:36.850074 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:36.850446 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:24:36.851116 (+ 670us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:24:36.851218 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:36.851279 (+ 61us) server_negotiation.cc:299] Negotiation successful
0807 02:24:36.851348 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 02:24:36.860363 19338 raft_consensus.cc:1261] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103 [term 1 FOLLOWER]: Refusing update from remote peer f34d7c5265af4d91b341795bcf5f9f1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:24:36.860363 19478 raft_consensus.cc:1261] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e [term 1 FOLLOWER]: Refusing update from remote peer f34d7c5265af4d91b341795bcf5f9f1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:24:36.861490 19699 consensus_queue.cc:1035] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:24:36.861794 19702 consensus_queue.cc:1035] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:24:36.883921 19180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:24:36.884202 19180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:24:36.887421 19180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: Sending DeleteTablet for 3 replicas of tablet d95bd793855e48aab516a5c990e1fa02
I20260807 02:24:36.888325 19601 tablet_service.cc:1514] Processing DeleteTablet for tablet d95bd793855e48aab516a5c990e1fa02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:24:36 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:57806
I20260807 02:24:36.888576 19458 tablet_service.cc:1514] Processing DeleteTablet for tablet d95bd793855e48aab516a5c990e1fa02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:24:36 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:39278
I20260807 02:24:36.888568 19316 tablet_service.cc:1514] Processing DeleteTablet for tablet d95bd793855e48aab516a5c990e1fa02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:24:36 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:39178
I20260807 02:24:36.889320 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 19250
I20260807 02:24:36.890661 19715 tablet_replica.cc:330] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a: stopping tablet replica
I20260807 02:24:36.891069 19715 raft_consensus.cc:2229] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:24:36.891607 19715 raft_consensus.cc:2258] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:24:36.893301 19715 ts_tablet_manager.cc:1905] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:24:36.896698 19715 ts_tablet_manager.cc:1918] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:24:36.896809 19715 log.cc:1196] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/d95bd793855e48aab516a5c990e1fa02
I20260807 02:24:36.897146 19715 ts_tablet_manager.cc:1939] T d95bd793855e48aab516a5c990e1fa02 P f34d7c5265af4d91b341795bcf5f9f1a: Deleting consensus metadata
I20260807 02:24:36.897301 19717 tablet_replica.cc:330] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e: stopping tablet replica
I20260807 02:24:36.897774 19717 raft_consensus.cc:2229] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:24:36.898331 19717 raft_consensus.cc:2258] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:24:36.898828 19167 catalog_manager.cc:4841] TS f34d7c5265af4d91b341795bcf5f9f1a (127.4.221.67:41487): tablet d95bd793855e48aab516a5c990e1fa02 (table test-table [id=f771321e85764048baa9c6da3a2a428e]) successfully deleted
I20260807 02:24:36.900245 19717 ts_tablet_manager.cc:1905] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 02:24:36.901234 19170 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:38365 (error 108)
I20260807 02:24:36.902328 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 19393
I20260807 02:24:36.902617 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.902221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:38365 (local address 127.0.0.1:39182)
0807 02:24:36.902398 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:36.902490 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260807 02:24:36.902792 19170 catalog_manager.cc:4584] TS 197797ffdacf4f3cb08d28ad7b071103 (127.4.221.65:38365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d95bd793855e48aab516a5c990e1fa02: Network error: Client connection negotiation failed: client connection to 127.4.221.65:38365: connect: Connection refused (error 111)
I20260807 02:24:36.911476 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 19531
I20260807 02:24:36.913841 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:36.911785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:33005 (local address 127.0.0.1:39290)
0807 02:24:36.913586 (+ 1801us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:36.913675 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.66:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1667}
I20260807 02:24:36.919235 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 19151
I20260807 02:24:36.928325 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44123
--webserver_interface=127.4.221.126
--webserver_port=37975
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:44123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:37.062317 19719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:37.062749 19719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:37.062829 19719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:37.067283 19719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:24:37.067397 19719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:37.067433 19719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:37.067472 19719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:24:37.067533 19719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:24:37.073575 19719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:44123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=37975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:37.075167 19719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:37.076547 19719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:37.093253 19730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:37.093259 19725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:37.093484 19719 server_base.cc:1034] running on GCE node
W20260807 02:24:37.093259 19726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:37.094417 19719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:37.096325 19719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:37.097587 19719 hybrid_clock.cc:648] HybridClock initialized: now 1786069477097567 us; error 38 us; skew 500 ppm
Aug 07 02:24:37 dist-test-slave-69rg krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069477, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:37.106977 19719 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:24:37.108246 19719 webserver.cc:460] Webserver started at http://127.4.221.126:37975/ using document root <none> and password file <none>
I20260807 02:24:37.108983 19719 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:37.109069 19719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:37.114205 19719 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260807 02:24:37.117198 19738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:37.119012 19719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:24:37.119199 19719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "4b369e37b60c4060b96afbf56d850544"
format_stamp: "Formatted at 2026-08-07 02:24:35 on dist-test-slave-69rg"
server_key: "8af10ef8c4415e0fad84388990f613c0"
server_key_iv: "51ef8c0a8b574de1e47d79f7f4640137"
server_key_version: "encryptionkey@0"
I20260807 02:24:37.120116 19719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:37.135888 19719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:37.140125 19719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:37.140385 19719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:37.149914 19719 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:44123
I20260807 02:24:37.150413 19793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:44123 every 8 connection(s)
I20260807 02:24:37.151508 19719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:24:37.157195 19794 sys_catalog.cc:263] Verifying existing consensus state
I20260807 02:24:37.157859 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 19719
I20260807 02:24:37.158411 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:38365
--local_ip_for_outbound_sockets=127.4.221.65
--tserver_master_addrs=127.4.221.126:44123
--webserver_port=45795
--webserver_interface=127.4.221.65
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:24:37.159718 19794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: Bootstrap starting.
I20260807 02:24:37.187214 19794 log.cc:824] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:37.199329 19794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:24:37.199819 19794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: Bootstrap complete.
I20260807 02:24:37.204952 19794 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } }
I20260807 02:24:37.205462 19794 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b369e37b60c4060b96afbf56d850544, State: Initialized, Role: FOLLOWER
I20260807 02:24:37.206080 19794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } }
I20260807 02:24:37.206204 19794 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:24:37.206302 19794 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:24:37.206424 19794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 1 FOLLOWER]: Advancing to term 2
I20260807 02:24:37.208593 19794 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } }
I20260807 02:24:37.208997 19794 leader_election.cc:302] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b369e37b60c4060b96afbf56d850544; no voters:
I20260807 02:24:37.209357 19794 leader_election.cc:288] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 02:24:37.209497 19798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 02:24:37.209771 19798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 2 LEADER]: Becoming Leader. State: Replica: 4b369e37b60c4060b96afbf56d850544, State: Running, Role: LEADER
I20260807 02:24:37.210197 19798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } }
I20260807 02:24:37.210821 19794 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:24:37.211846 19800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b369e37b60c4060b96afbf56d850544. Latest consensus state: current_term: 2 leader_uuid: "4b369e37b60c4060b96afbf56d850544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } } }
I20260807 02:24:37.212002 19800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:37.212402 19799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4b369e37b60c4060b96afbf56d850544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } } }
I20260807 02:24:37.212503 19799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:37.212590 19807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:24:37.218261 19807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:24:37.218979 19807 catalog_manager.cc:1259] Loaded cluster ID: 63b8648bcf624c6abf5e0ce4d3458459
I20260807 02:24:37.219077 19807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:24:37.220557 19807 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260807 02:24:37.225443 19807 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260807 02:24:37.231639 19807 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:24:37.232537 19807 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: Loaded TSK: 0
I20260807 02:24:37.233834 19807 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 02:24:37.304705 19796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:37.304988 19796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:37.305034 19796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:37.309252 19796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:37.309320 19796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:37.309417 19796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:24:37.315362 19796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:38365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=45795
--tserver_master_addrs=127.4.221.126:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:37.316558 19796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:37.317698 19796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:37.329021 19822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:37.329016 19821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:37.329016 19824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:37.329452 19796 server_base.cc:1034] running on GCE node
I20260807 02:24:37.330065 19796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:37.330976 19796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:37.332203 19796 hybrid_clock.cc:648] HybridClock initialized: now 1786069477332173 us; error 49 us; skew 500 ppm
Aug 07 02:24:37 dist-test-slave-69rg krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069477, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:37.336024 19796 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:24:37.337311 19796 webserver.cc:460] Webserver started at http://127.4.221.65:45795/ using document root <none> and password file <none>
I20260807 02:24:37.338171 19796 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:37.338290 19796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:37.343626 19796 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 02:24:37.346294 19832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:37.347838 19796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:24:37.347958 19796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "197797ffdacf4f3cb08d28ad7b071103"
format_stamp: "Formatted at 2026-08-07 02:24:35 on dist-test-slave-69rg"
server_key: "7f167da237ac21a3a4fed6b424367627"
server_key_iv: "e47c55ef4712599d955ec02fcb703078"
server_key_version: "encryptionkey@0"
I20260807 02:24:37.348673 19796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:37.373409 19796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:37.377274 19796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:37.377507 19796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:37.378276 19796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:37.379423 19839 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 02:24:37.383929 19796 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 02:24:37.384035 19796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260807 02:24:37.384115 19796 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 02:24:37.385970 19796 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 02:24:37.386040 19796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260807 02:24:37.386354 19839 tablet_bootstrap.cc:492] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: Bootstrap starting.
I20260807 02:24:37.397166 19796 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:38365
I20260807 02:24:37.397241 19954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:38365 every 8 connection(s)
I20260807 02:24:37.398368 19796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:24:37.398702 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 19796
I20260807 02:24:37.399248 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:33005
--local_ip_for_outbound_sockets=127.4.221.66
--tserver_master_addrs=127.4.221.126:44123
--webserver_port=32851
--webserver_interface=127.4.221.66
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:24:37.406949 19839 log.cc:824] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: Log is configured to *not* fsync() on all Append() calls
Aug 07 02:24:37 dist-test-slave-69rg krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069477, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:37.416716 19839 tablet_bootstrap.cc:492] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:24:37.417232 19839 tablet_bootstrap.cc:492] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: Bootstrap complete.
I20260807 02:24:37.419106 19839 ts_tablet_manager.cc:1397] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.020s
I20260807 02:24:37.420284 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.400949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:59301 (local address 127.4.221.126:44123)
0807 02:24:37.402154 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:37.402176 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:37.402230 (+ 54us) server_negotiation.cc:407] Connection header received
0807 02:24:37.403451 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:37.403464 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:37.403786 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:37.404063 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:37.405735 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.407496 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.408940 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.409810 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.410057 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.413651 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:37.413658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:37.413676 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:37.413725 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:37.416077 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:37.416818 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:37.416825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:37.416831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:37.416998 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:37.417450 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:37.417453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:37.417455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:37.417986 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:37.418102 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:37.418902 (+ 800us) server_negotiation.cc:299] Negotiation successful
0807 02:24:37.419275 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1025,"thread_start_us":120,"threads_started":1}
I20260807 02:24:37.421942 19960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.400941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:59301)
0807 02:24:37.401668 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:37.401748 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:37.402934 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:37.404201 (+ 1267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:37.404216 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:37.404856 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:37.405504 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.405525 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.407699 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.407702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.408788 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.408799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.410008 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.410015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.410218 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.410960 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:37.410984 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:37.413323 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:37.416268 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:37.416273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:37.416287 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:37.416667 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:37.417136 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:37.417140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:37.417142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:37.417329 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:37.418218 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:37.418227 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:37.418550 (+ 323us) client_negotiation.cc:769] Sending connection context
0807 02:24:37.420676 (+ 2126us) client_negotiation.cc:240] Negotiation successful
0807 02:24:37.421013 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":166,"threads_started":1}
I20260807 02:24:37.422195 19839 raft_consensus.cc:348] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } }
I20260807 02:24:37.422776 19839 raft_consensus.cc:729] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197797ffdacf4f3cb08d28ad7b071103, State: Initialized, Role: FOLLOWER
I20260807 02:24:37.423276 19839 consensus_queue.cc:260] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } }
I20260807 02:24:37.424371 19839 ts_tablet_manager.cc:1428] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260807 02:24:37.424787 19955 heartbeater.cc:347] Connected to a master server at 127.4.221.126:44123
I20260807 02:24:37.425027 19955 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:37.425622 19955 heartbeater.cc:511] Master 127.4.221.126:44123 requested a full tablet report, sending...
I20260807 02:24:37.428026 19756 ts_manager.cc:194] Registered new tserver with Master: 197797ffdacf4f3cb08d28ad7b071103 (127.4.221.65:38365)
I20260807 02:24:37.431614 19756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:59301
I20260807 02:24:37.434242 19955 heartbeater.cc:503] Master 127.4.221.126:44123 was elected leader, sending a full tablet report...
Aug 07 02:24:37 dist-test-slave-69rg krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069477, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for kudu/127.4.221.65@KRBTEST.COM
I20260807 02:24:37.443123 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.430919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:38365 (local address 127.0.0.1:39184)
0807 02:24:37.431737 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:37.431793 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:37.431959 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:37.432611 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:37.432616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:37.433053 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:37.433201 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.433209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.434555 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.434558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.435507 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.435519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.436270 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.436274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.436393 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.437023 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:37.437040 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:37.438930 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:37.441109 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:37.441117 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:37.441122 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:37.441382 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:37.441743 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:37.441746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:37.441747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:37.441912 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:37.442506 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:37.442513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:37.442654 (+ 141us) client_negotiation.cc:769] Sending connection context
0807 02:24:37.442806 (+ 152us) client_negotiation.cc:240] Negotiation successful
0807 02:24:37.442980 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":128,"threads_started":1}
I20260807 02:24:37.443277 19964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.431028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39184 (local address 127.4.221.65:38365)
0807 02:24:37.431467 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:37.431479 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:37.431913 (+ 434us) server_negotiation.cc:407] Connection header received
0807 02:24:37.432116 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:37.432122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:37.432327 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:37.432480 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:37.433344 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.434417 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.435648 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.436141 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.436232 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.439088 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:37.439099 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:37.439112 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:37.439152 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:37.440981 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:37.441508 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:37.441515 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:37.441517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:37.441615 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:37.442009 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:37.442012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:37.442014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:37.442361 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:37.442498 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:37.442917 (+ 419us) server_negotiation.cc:299] Negotiation successful
0807 02:24:37.443101 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":142,"threads_started":1}
I20260807 02:24:37.444326 19883 tablet_service.cc:1514] Processing DeleteTablet for tablet d95bd793855e48aab516a5c990e1fa02 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:24:36 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:39184
I20260807 02:24:37.445281 19965 tablet_replica.cc:330] stopping tablet replica
I20260807 02:24:37.445627 19965 raft_consensus.cc:2229] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:24:37.445936 19965 raft_consensus.cc:2258] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:24:37.447166 19965 ts_tablet_manager.cc:1905] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:24:37.452286 19965 ts_tablet_manager.cc:1918] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:24:37.452410 19965 log.cc:1196] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal/wals/d95bd793855e48aab516a5c990e1fa02
I20260807 02:24:37.452715 19965 ts_tablet_manager.cc:1939] T d95bd793855e48aab516a5c990e1fa02 P 197797ffdacf4f3cb08d28ad7b071103: Deleting consensus metadata
I20260807 02:24:37.455209 19742 catalog_manager.cc:4841] TS 197797ffdacf4f3cb08d28ad7b071103 (127.4.221.65:38365): tablet d95bd793855e48aab516a5c990e1fa02 (table test-table [id=f771321e85764048baa9c6da3a2a428e]) successfully deleted
W20260807 02:24:37.527364 19958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:37.527618 19958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:37.527669 19958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:37.531620 19958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:37.531679 19958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:37.531736 19958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:24:37.536872 19958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:33005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=32851
--tserver_master_addrs=127.4.221.126:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:37.537912 19958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:37.538903 19958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:37.548516 19973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:37.548538 19971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:37.548516 19970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:37.549021 19958 server_base.cc:1034] running on GCE node
I20260807 02:24:37.549396 19958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:37.550127 19958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:37.551301 19958 hybrid_clock.cc:648] HybridClock initialized: now 1786069477551257 us; error 48 us; skew 500 ppm
Aug 07 02:24:37 dist-test-slave-69rg krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069477, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:37.554309 19958 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:24:37.555239 19958 webserver.cc:460] Webserver started at http://127.4.221.66:32851/ using document root <none> and password file <none>
I20260807 02:24:37.556021 19958 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:37.556249 19958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:37.561074 19958 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:24:37.563602 19980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:37.564980 19958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:24:37.565165 19958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "631ebf3e199c415abdc52720f90bc90e"
format_stamp: "Formatted at 2026-08-07 02:24:36 on dist-test-slave-69rg"
server_key: "bfd3023565641e90f6fc7bcd21e548c4"
server_key_iv: "4d1c086f9c457fb89870afdb2a0113dd"
server_key_version: "encryptionkey@0"
I20260807 02:24:37.565752 19958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:37.577579 19958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:37.580649 19958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:37.580865 19958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:37.581470 19958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:37.582635 19988 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260807 02:24:37.586419 19988 ts_tablet_manager.cc:1852] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260807 02:24:37.586848 19988 ts_tablet_manager.cc:1905] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:24:37.590840 19988 ts_tablet_manager.cc:1918] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:24:37.590981 19988 log.cc:1196] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/d95bd793855e48aab516a5c990e1fa02
I20260807 02:24:37.591341 19988 ts_tablet_manager.cc:1939] T d95bd793855e48aab516a5c990e1fa02 P 631ebf3e199c415abdc52720f90bc90e: Deleting consensus metadata
I20260807 02:24:37.591657 19958 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260807 02:24:37.591719 19958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260807 02:24:37.591810 19958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:24:37.591836 19958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:37.601287 19958 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:33005
I20260807 02:24:37.601358 20099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:33005 every 8 connection(s)
I20260807 02:24:37.602293 19958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:24:37.609690 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 19958
I20260807 02:24:37.610160 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:41487
--local_ip_for_outbound_sockets=127.4.221.67
--tserver_master_addrs=127.4.221.126:44123
--webserver_port=44153
--webserver_interface=127.4.221.67
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:24:37 dist-test-slave-69rg krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069477, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:37.616897 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.604114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:36339 (local address 127.4.221.126:44123)
0807 02:24:37.604327 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:37.604330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:37.605170 (+ 840us) server_negotiation.cc:407] Connection header received
0807 02:24:37.606318 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:37.606321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:37.606372 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:37.606447 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:37.607593 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.608134 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.609255 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.609602 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.609732 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.612678 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:37.612693 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:37.612695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:37.612719 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:37.614503 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:37.614997 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:37.615003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:37.615008 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:37.615104 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:37.615477 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:37.615484 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:37.615488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:37.615786 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:37.615901 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:37.616467 (+ 566us) server_negotiation.cc:299] Negotiation successful
0807 02:24:37.616638 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 02:24:37.617627 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.604497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:36339)
0807 02:24:37.604994 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:37.605039 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:37.606058 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:37.606607 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:37.606614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:37.606975 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:37.607451 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.607464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.608260 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.608263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.609136 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.609146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.609794 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.609799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.610004 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.610695 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:37.610759 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:37.612514 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:37.614616 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:37.614621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:37.614644 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:37.614893 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:37.615225 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:37.615228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:37.615229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:37.615368 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:37.615941 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:37.615948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:37.616210 (+ 262us) client_negotiation.cc:769] Sending connection context
0807 02:24:37.616376 (+ 166us) client_negotiation.cc:240] Negotiation successful
0807 02:24:37.616715 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":127,"threads_started":1}
I20260807 02:24:37.618942 20100 heartbeater.cc:347] Connected to a master server at 127.4.221.126:44123
I20260807 02:24:37.619184 20100 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:37.619692 20100 heartbeater.cc:511] Master 127.4.221.126:44123 requested a full tablet report, sending...
I20260807 02:24:37.620810 19756 ts_manager.cc:194] Registered new tserver with Master: 631ebf3e199c415abdc52720f90bc90e (127.4.221.66:33005)
I20260807 02:24:37.621701 19756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:36339
W20260807 02:24:37.727945 20103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:37.728267 20103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:37.728387 20103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:37.732610 20103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:37.732689 20103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:37.732752 20103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:24:37.738548 20103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:41487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=44153
--tserver_master_addrs=127.4.221.126:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:37.739799 20103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:37.740865 20103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:37.750571 20109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:37.750572 20111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:37.750780 20103 server_base.cc:1034] running on GCE node
W20260807 02:24:37.750588 20108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:37.751360 20103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:37.752012 20103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:37.753172 20103 hybrid_clock.cc:648] HybridClock initialized: now 1786069477753144 us; error 43 us; skew 500 ppm
Aug 07 02:24:37 dist-test-slave-69rg krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069477, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:37.756129 20103 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:24:37.757031 20103 webserver.cc:460] Webserver started at http://127.4.221.67:44153/ using document root <none> and password file <none>
I20260807 02:24:37.757594 20103 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:37.757691 20103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:37.761601 20103 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:24:37.763748 20118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:37.765034 20103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:24:37.765198 20103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "f34d7c5265af4d91b341795bcf5f9f1a"
format_stamp: "Formatted at 2026-08-07 02:24:36 on dist-test-slave-69rg"
server_key: "85a138e2ed3a42ba1f058677ccbd56f8"
server_key_iv: "bb9635437cf0e524c607be9f9cd9c66e"
server_key_version: "encryptionkey@0"
I20260807 02:24:37.765777 20103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:37.780207 20103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:37.783303 20103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:37.783524 20103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:37.784152 20103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:37.784993 20103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:24:37.785065 20103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:37.785138 20103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:24:37.785189 20103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:37.794294 20103 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:41487
I20260807 02:24:37.794318 20241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:41487 every 8 connection(s)
I20260807 02:24:37.795312 20103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:24:37.799346 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 20103
Aug 07 02:24:37 dist-test-slave-69rg krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069477, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:37.809990 20245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.800147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37568 (local address 127.4.221.126:44123)
0807 02:24:37.800454 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:37.800459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:37.800820 (+ 361us) server_negotiation.cc:407] Connection header received
0807 02:24:37.801016 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:37.801020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:37.801082 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:37.801238 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:37.802188 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.802803 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.803786 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.804047 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.804169 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.805935 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:37.805939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:37.805941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:37.805963 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:37.808014 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:37.808513 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:37.808519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:37.808521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:37.808637 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:37.808973 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:37.808976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:37.808977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:37.809169 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:37.809364 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:37.809594 (+ 230us) server_negotiation.cc:299] Negotiation successful
0807 02:24:37.809733 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":91,"threads_started":1}
I20260807 02:24:37.811327 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.797357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:57981 (local address 127.4.221.126:44123)
0807 02:24:37.797525 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:37.797530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:37.798501 (+ 971us) server_negotiation.cc:407] Connection header received
0807 02:24:37.800403 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:37.800406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:37.800471 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:37.800596 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:37.801858 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.802402 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.803634 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.803858 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.803967 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.806969 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:37.806992 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:37.806996 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:37.807041 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:37.808825 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:37.809385 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:37.809390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:37.809392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:37.809477 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:37.809983 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:37.809987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:37.809988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:37.810211 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:37.810310 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:37.811090 (+ 780us) server_negotiation.cc:299] Negotiation successful
0807 02:24:37.811213 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 02:24:37.812511 20244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.797795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:57981)
0807 02:24:37.798330 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:37.798374 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:37.799864 (+ 1490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:37.800804 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:37.800812 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:37.801186 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:37.801679 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.801692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.802550 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.802559 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.803496 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.803506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.804016 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.804020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.804206 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.804858 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:37.804881 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:37.806732 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:37.808937 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:37.808945 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:37.808963 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:37.809275 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:37.809605 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:37.809610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:37.809613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:37.809809 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:37.810378 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:37.810385 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:37.810772 (+ 387us) client_negotiation.cc:769] Sending connection context
0807 02:24:37.811002 (+ 230us) client_negotiation.cc:240] Negotiation successful
0807 02:24:37.811406 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":120,"threads_started":1}
I20260807 02:24:37.813688 20242 heartbeater.cc:347] Connected to a master server at 127.4.221.126:44123
I20260807 02:24:37.814047 20242 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:37.814647 20242 heartbeater.cc:511] Master 127.4.221.126:44123 requested a full tablet report, sending...
I20260807 02:24:37.815711 19756 ts_manager.cc:194] Registered new tserver with Master: f34d7c5265af4d91b341795bcf5f9f1a (127.4.221.67:41487)
I20260807 02:24:37.816311 19756 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:57981
I20260807 02:24:37.824119 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:24:37.828610 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.824852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37584 (local address 127.4.221.126:44123)
0807 02:24:37.824998 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:37.825003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:37.825274 (+ 271us) server_negotiation.cc:407] Connection header received
0807 02:24:37.825412 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:37.825414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:37.825530 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:37.825616 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:24:37.825925 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.826426 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.827291 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.827496 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.827571 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.827839 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:24:37.828236 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:24:37.828323 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:37.828389 (+ 66us) server_negotiation.cc:299] Negotiation successful
0807 02:24:37.828435 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 02:24:37.829289 19756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:24:37.832664 19756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:24:37.840611 19883 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3dda852e2b4bec96b00622ecd7bf14 (DEFAULT_TABLE table=test-table [id=74636230a423493983bea1f3d5016e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:24:37.841338 19883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3dda852e2b4bec96b00622ecd7bf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:37.844322 19839 tablet_bootstrap.cc:492] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103: Bootstrap starting.
I20260807 02:24:37.845189 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.839405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:41487 (local address 127.0.0.1:57814)
0807 02:24:37.839657 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:37.839676 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:37.839755 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:37.840696 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:37.840699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:37.840718 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:37.840842 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.840852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.842186 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.842192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.843539 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.843551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.844714 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.844718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.844819 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.844863 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 02:24:37.844944 (+ 81us) client_negotiation.cc:240] Negotiation successful
0807 02:24:37.845033 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":243}
I20260807 02:24:37.845348 19839 tablet_bootstrap.cc:654] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:37.845412 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.839567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:33005 (local address 127.0.0.1:39296)
0807 02:24:37.840027 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:37.840041 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:37.840122 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:37.840876 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:37.840879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:37.840898 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:37.840992 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.841001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.842876 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.842880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.843877 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.843886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.844925 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.844929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.845019 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.845044 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 02:24:37.845095 (+ 51us) client_negotiation.cc:240] Negotiation successful
0807 02:24:37.845185 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":8,"thread_start_us":136,"threads_started":1}
I20260807 02:24:37.845886 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.839682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.4.221.67:41487)
0807 02:24:37.840220 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:37.840232 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:37.840257 (+ 25us) server_negotiation.cc:407] Connection header received
0807 02:24:37.840347 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:37.840351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:37.840524 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:37.840708 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:37.840982 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.842015 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.843689 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.844561 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.844708 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.844865 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:37.845576 (+ 711us) server_negotiation.cc:299] Negotiation successful
0807 02:24:37.845690 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":180,"threads_started":1}
I20260807 02:24:37.846149 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.840005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.4.221.66:33005)
0807 02:24:37.840411 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:37.840422 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:37.840445 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:24:37.840517 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:37.840521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:37.840708 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:37.841146 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:37.841323 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.842720 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.843997 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.844760 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.845589 (+ 829us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.845764 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:37.845862 (+ 98us) server_negotiation.cc:299] Negotiation successful
0807 02:24:37.845958 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":152,"threads_started":1}
I20260807 02:24:37.847411 19839 tablet_bootstrap.cc:492] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103: No bootstrap required, opened a new log
I20260807 02:24:37.847518 19839 ts_tablet_manager.cc:1397] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260807 02:24:37.847515 20031 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3dda852e2b4bec96b00622ecd7bf14 (DEFAULT_TABLE table=test-table [id=74636230a423493983bea1f3d5016e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:24:37.847986 19839 raft_consensus.cc:348] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:37.848105 19839 raft_consensus.cc:374] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:37.848170 19839 raft_consensus.cc:729] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197797ffdacf4f3cb08d28ad7b071103, State: Initialized, Role: FOLLOWER
I20260807 02:24:37.848304 19839 consensus_queue.cc:260] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:37.848591 19839 ts_tablet_manager.cc:1428] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 02:24:37.848699 20031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3dda852e2b4bec96b00622ecd7bf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:37.849117 20169 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3dda852e2b4bec96b00622ecd7bf14 (DEFAULT_TABLE table=test-table [id=74636230a423493983bea1f3d5016e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:24:37.850271 20169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3dda852e2b4bec96b00622ecd7bf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:37.853952 19988 tablet_bootstrap.cc:492] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e: Bootstrap starting.
I20260807 02:24:37.856298 19988 tablet_bootstrap.cc:654] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:37.857239 19988 log.cc:824] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:37.858327 20257 tablet_bootstrap.cc:492] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: Bootstrap starting.
I20260807 02:24:37.859601 19988 tablet_bootstrap.cc:492] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e: No bootstrap required, opened a new log
I20260807 02:24:37.859817 19988 ts_tablet_manager.cc:1397] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 02:24:37.860657 20257 tablet_bootstrap.cc:654] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:37.861449 20257 log.cc:824] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:37.863386 20257 tablet_bootstrap.cc:492] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: No bootstrap required, opened a new log
I20260807 02:24:37.863304 19988 raft_consensus.cc:348] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:37.863605 19988 raft_consensus.cc:374] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:37.863631 20257 ts_tablet_manager.cc:1397] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 02:24:37.863677 19988 raft_consensus.cc:729] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 631ebf3e199c415abdc52720f90bc90e, State: Initialized, Role: FOLLOWER
I20260807 02:24:37.864109 19988 consensus_queue.cc:260] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:37.864794 20100 heartbeater.cc:503] Master 127.4.221.126:44123 was elected leader, sending a full tablet report...
I20260807 02:24:37.865176 19988 ts_tablet_manager.cc:1428] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260807 02:24:37.867007 20257 raft_consensus.cc:348] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:37.867272 20257 raft_consensus.cc:374] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:37.867358 20257 raft_consensus.cc:729] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f34d7c5265af4d91b341795bcf5f9f1a, State: Initialized, Role: FOLLOWER
I20260807 02:24:37.867849 20257 consensus_queue.cc:260] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:37.868593 20242 heartbeater.cc:503] Master 127.4.221.126:44123 was elected leader, sending a full tablet report...
I20260807 02:24:37.868846 20257 ts_tablet_manager.cc:1428] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260807 02:24:37.900058 19957 tablet.cc:2369] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:24:37.937808 20259 raft_consensus.cc:482] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:24:37.938045 20259 raft_consensus.cc:504] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:37.939219 20259 leader_election.cc:288] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f34d7c5265af4d91b341795bcf5f9f1a (127.4.221.67:41487), 197797ffdacf4f3cb08d28ad7b071103 (127.4.221.65:38365)
I20260807 02:24:37.943707 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.939651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:41487 (local address 127.4.221.66:40471)
0807 02:24:37.939788 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:37.939805 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:37.939906 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:37.940214 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:37.940217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:37.940235 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:37.940393 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.940404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.941389 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.941393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.942372 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.942382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.943241 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.943245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.943361 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.943385 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 02:24:37.943474 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 02:24:37.943545 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"spinlock_wait_cycles":3840}
I20260807 02:24:37.943697 20252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.939752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:40471 (local address 127.4.221.67:41487)
0807 02:24:37.939888 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:37.939893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:37.939912 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:24:37.940012 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:37.940015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:37.940071 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:37.940191 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:37.940539 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.941267 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.942509 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.943090 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.943185 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.943225 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:37.943516 (+ 291us) server_negotiation.cc:299] Negotiation successful
0807 02:24:37.943585 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 02:24:37.944361 19964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.939845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:55479 (local address 127.4.221.65:38365)
0807 02:24:37.939994 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:37.940000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:37.940256 (+ 256us) server_negotiation.cc:407] Connection header received
0807 02:24:37.940468 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:37.940474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:37.940554 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:37.940681 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:37.940989 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.941713 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.942977 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.943658 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.943787 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.943958 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:37.944093 (+ 135us) server_negotiation.cc:299] Negotiation successful
0807 02:24:37.944240 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 02:24:37.944420 20192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3dda852e2b4bec96b00622ecd7bf14" candidate_uuid: "631ebf3e199c415abdc52720f90bc90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" is_pre_election: true
I20260807 02:24:37.944802 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.939669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:38365 (local address 127.4.221.66:55479)
0807 02:24:37.940158 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:37.940168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:37.940300 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:37.940712 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:37.940715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:37.940733 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:37.940854 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.940860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.941882 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.941885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.942837 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:37.942848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.943801 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.943805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:37.943932 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.943957 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 02:24:37.944573 (+ 616us) client_negotiation.cc:240] Negotiation successful
0807 02:24:37.944654 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":128,"spinlock_wait_cycles":21632,"thread_start_us":139,"threads_started":1}
I20260807 02:24:37.944766 20192 raft_consensus.cc:2454] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 631ebf3e199c415abdc52720f90bc90e in term 0.
I20260807 02:24:37.945325 19981 leader_election.cc:302] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 631ebf3e199c415abdc52720f90bc90e, f34d7c5265af4d91b341795bcf5f9f1a; no voters:
I20260807 02:24:37.945465 19905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3dda852e2b4bec96b00622ecd7bf14" candidate_uuid: "631ebf3e199c415abdc52720f90bc90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197797ffdacf4f3cb08d28ad7b071103" is_pre_election: true
I20260807 02:24:37.945624 20259 raft_consensus.cc:2781] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:24:37.945722 20259 raft_consensus.cc:482] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:24:37.945777 20259 raft_consensus.cc:3037] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:37.945801 19905 raft_consensus.cc:2454] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 631ebf3e199c415abdc52720f90bc90e in term 0.
I20260807 02:24:37.947026 20259 raft_consensus.cc:504] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:37.947358 20259 leader_election.cc:288] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [CANDIDATE]: Term 1 election: Requested vote from peers f34d7c5265af4d91b341795bcf5f9f1a (127.4.221.67:41487), 197797ffdacf4f3cb08d28ad7b071103 (127.4.221.65:38365)
I20260807 02:24:37.947691 20192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3dda852e2b4bec96b00622ecd7bf14" candidate_uuid: "631ebf3e199c415abdc52720f90bc90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f34d7c5265af4d91b341795bcf5f9f1a"
I20260807 02:24:37.947702 19905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3dda852e2b4bec96b00622ecd7bf14" candidate_uuid: "631ebf3e199c415abdc52720f90bc90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "197797ffdacf4f3cb08d28ad7b071103"
I20260807 02:24:37.947825 20192 raft_consensus.cc:3037] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:37.947837 19905 raft_consensus.cc:3037] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:37.948791 19905 raft_consensus.cc:2454] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 631ebf3e199c415abdc52720f90bc90e in term 1.
I20260807 02:24:37.948938 20192 raft_consensus.cc:2454] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 631ebf3e199c415abdc52720f90bc90e in term 1.
I20260807 02:24:37.949091 19985 leader_election.cc:302] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197797ffdacf4f3cb08d28ad7b071103, 631ebf3e199c415abdc52720f90bc90e; no voters:
I20260807 02:24:37.949316 20259 raft_consensus.cc:2781] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:24:37.949532 20259 raft_consensus.cc:686] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 1 LEADER]: Becoming Leader. State: Replica: 631ebf3e199c415abdc52720f90bc90e, State: Running, Role: LEADER
I20260807 02:24:37.949910 20259 consensus_queue.cc:237] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:37.952973 19756 catalog_manager.cc:5495] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e reported cstate change: term changed from 0 to 1, leader changed from <none> to 631ebf3e199c415abdc52720f90bc90e (127.4.221.66). New cstate: current_term: 1 leader_uuid: "631ebf3e199c415abdc52720f90bc90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } health_report { overall_health: UNKNOWN } } }
I20260807 02:24:37.973742 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:37.970216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.4.221.66:33005)
0807 02:24:37.970354 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:37.970358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:37.970411 (+ 53us) server_negotiation.cc:407] Connection header received
0807 02:24:37.970532 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:37.970535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:37.970593 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:37.970688 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:24:37.970902 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.971424 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.972262 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:37.972484 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:37.972573 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:37.972904 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:24:37.973350 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:24:37.973443 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:37.973575 (+ 132us) server_negotiation.cc:299] Negotiation successful
0807 02:24:37.973628 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 02:24:37.979471 20192 raft_consensus.cc:1261] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a [term 1 FOLLOWER]: Refusing update from remote peer 631ebf3e199c415abdc52720f90bc90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:24:37.979470 19905 raft_consensus.cc:1261] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 1 FOLLOWER]: Refusing update from remote peer 631ebf3e199c415abdc52720f90bc90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:24:37.980154 20259 consensus_queue.cc:1035] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:24:37.980392 20262 consensus_queue.cc:1035] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:24:37.987217 20265 mvcc.cc:205] Tried to move back new op lower bound from 7315740581796438016 to 7315740581685170176. Current Snapshot: MvccSnapshot[applied={T|T < 7315740581796438016}]
I20260807 02:24:37.989007 20267 mvcc.cc:205] Tried to move back new op lower bound from 7315740581796438016 to 7315740581685170176. Current Snapshot: MvccSnapshot[applied={T|T < 7315740581796438016}]
I20260807 02:24:37.995407 19755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:24:37.995553 19755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:24:37.997983 19755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: Sending DeleteTablet for 3 replicas of tablet 8c3dda852e2b4bec96b00622ecd7bf14
I20260807 02:24:37.998750 19883 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3dda852e2b4bec96b00622ecd7bf14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:24:37 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:39184
I20260807 02:24:37.998703 20169 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3dda852e2b4bec96b00622ecd7bf14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:24:37 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:57814
I20260807 02:24:37.998831 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 19796
I20260807 02:24:38.006412 20274 tablet_replica.cc:330] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: stopping tablet replica
I20260807 02:24:38.006394 20031 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3dda852e2b4bec96b00622ecd7bf14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:24:37 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:39296
I20260807 02:24:38.006814 20274 raft_consensus.cc:2229] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:24:38.007187 20274 raft_consensus.cc:2258] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:24:38.007112 20275 tablet_replica.cc:330] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e: stopping tablet replica
I20260807 02:24:38.007597 20275 raft_consensus.cc:2229] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:24:38.008292 20275 raft_consensus.cc:2258] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:24:38.009655 20274 ts_tablet_manager.cc:1905] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260807 02:24:38.009763 19742 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:38365 (error 108)
I20260807 02:24:38.010953 20275 ts_tablet_manager.cc:1905] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:24:38.011174 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 19958
I20260807 02:24:38.013073 20274 ts_tablet_manager.cc:1918] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:24:38.013207 20274 log.cc:1196] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/8c3dda852e2b4bec96b00622ecd7bf14
I20260807 02:24:38.013576 20274 ts_tablet_manager.cc:1939] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: Deleting consensus metadata
I20260807 02:24:38.014165 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:38.013481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:38365 (local address 127.0.0.1:39186)
0807 02:24:38.013640 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:38.013731 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260807 02:24:38.014400 19742 catalog_manager.cc:4584] TS 197797ffdacf4f3cb08d28ad7b071103 (127.4.221.65:38365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c3dda852e2b4bec96b00622ecd7bf14: Network error: Client connection negotiation failed: client connection to 127.4.221.65:38365: connect: Connection refused (error 111)
I20260807 02:24:38.015141 19739 catalog_manager.cc:4841] TS f34d7c5265af4d91b341795bcf5f9f1a (127.4.221.67:41487): tablet 8c3dda852e2b4bec96b00622ecd7bf14 (table test-table [id=74636230a423493983bea1f3d5016e93]) successfully deleted
I20260807 02:24:38.019172 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 20103
I20260807 02:24:38.021032 20251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:38.019318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:33005 (local address 127.0.0.1:39316)
0807 02:24:38.020703 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:38.020924 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.66:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1267}
I20260807 02:24:38.027114 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 19719
I20260807 02:24:38.035131 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44123
--webserver_interface=127.4.221.126
--webserver_port=37975
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:44123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:38.152679 20277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:38.152966 20277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:38.153009 20277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:38.157043 20277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:24:38.157105 20277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:38.157125 20277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:38.157142 20277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:24:38.157158 20277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:24:38.162520 20277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:44123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=37975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:38.163599 20277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:38.164582 20277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:38.173091 20283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:38.173149 20277 server_base.cc:1034] running on GCE node
W20260807 02:24:38.173098 20285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:38.173098 20282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:38.174029 20277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:38.175180 20277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:38.176369 20277 hybrid_clock.cc:648] HybridClock initialized: now 1786069478176343 us; error 38 us; skew 500 ppm
Aug 07 02:24:38 dist-test-slave-69rg krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069478, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:38.179292 20277 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:24:38.180207 20277 webserver.cc:460] Webserver started at http://127.4.221.126:37975/ using document root <none> and password file <none>
I20260807 02:24:38.180781 20277 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:38.180877 20277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:38.184769 20277 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:24:38.186980 20292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:38.188295 20277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 02:24:38.188437 20277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "4b369e37b60c4060b96afbf56d850544"
format_stamp: "Formatted at 2026-08-07 02:24:35 on dist-test-slave-69rg"
server_key: "8af10ef8c4415e0fad84388990f613c0"
server_key_iv: "51ef8c0a8b574de1e47d79f7f4640137"
server_key_version: "encryptionkey@0"
I20260807 02:24:38.189015 20277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:38.226073 20277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:38.229696 20277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:38.229954 20277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:38.237548 20277 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:44123
I20260807 02:24:38.237912 20346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:44123 every 8 connection(s)
I20260807 02:24:38.238760 20277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:24:38.242887 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 20277
I20260807 02:24:38.243152 20347 sys_catalog.cc:263] Verifying existing consensus state
I20260807 02:24:38.243425 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:38365
--local_ip_for_outbound_sockets=127.4.221.65
--tserver_master_addrs=127.4.221.126:44123
--webserver_port=45795
--webserver_interface=127.4.221.65
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:24:38.244995 20347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: Bootstrap starting.
I20260807 02:24:38.268708 20347 log.cc:824] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:38.280467 20347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:24:38.280936 20347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544: Bootstrap complete.
I20260807 02:24:38.285264 20347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } }
I20260807 02:24:38.285713 20347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b369e37b60c4060b96afbf56d850544, State: Initialized, Role: FOLLOWER
I20260807 02:24:38.286252 20347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } }
I20260807 02:24:38.286371 20347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:24:38.286463 20347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:24:38.286589 20347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 2 FOLLOWER]: Advancing to term 3
I20260807 02:24:38.288565 20347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } }
I20260807 02:24:38.288916 20347 leader_election.cc:302] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b369e37b60c4060b96afbf56d850544; no voters:
I20260807 02:24:38.289268 20347 leader_election.cc:288] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 02:24:38.289376 20351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 02:24:38.289604 20351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [term 3 LEADER]: Becoming Leader. State: Replica: 4b369e37b60c4060b96afbf56d850544, State: Running, Role: LEADER
I20260807 02:24:38.289933 20351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } }
I20260807 02:24:38.290566 20347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:24:38.291282 20352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4b369e37b60c4060b96afbf56d850544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } } }
I20260807 02:24:38.291406 20352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:38.292268 20359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:24:38.292495 20353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b369e37b60c4060b96afbf56d850544. Latest consensus state: current_term: 3 leader_uuid: "4b369e37b60c4060b96afbf56d850544" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b369e37b60c4060b96afbf56d850544" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44123 } } }
I20260807 02:24:38.292673 20353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b369e37b60c4060b96afbf56d850544 [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:38.297499 20359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:24:38.298247 20359 catalog_manager.cc:1259] Loaded cluster ID: 63b8648bcf624c6abf5e0ce4d3458459
I20260807 02:24:38.298313 20359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260807 02:24:38.299521 20359 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f974aefb9ad google::LogMessage::Fail() at ??:0
@ 0x7f974aefe88a google::LogMessage::SendToLog() at ??:0
@ 0x7f974aefb4de google::LogMessage::Flush() at ??:0
@ 0x7f974aefca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f975583afdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f975583bd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9755838579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f975587c640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f974d14d3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f974d14dc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f974d14f3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f974d13e512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f974e7826db start_thread at ??:0
@ 0x7f97499f961f clone at ??:0
W20260807 02:24:38.401733 20349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:38.402110 20349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:38.402211 20349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:38.406412 20349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:38.406500 20349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:38.406589 20349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:24:38.412271 20349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:38365
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=45795
--tserver_master_addrs=127.4.221.126:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:38.413476 20349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:38.414603 20349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:38.425686 20391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:38.425673 20390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:38.425873 20349 server_base.cc:1034] running on GCE node
W20260807 02:24:38.425719 20393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:38.426620 20349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:38.427331 20349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:38.428537 20349 hybrid_clock.cc:648] HybridClock initialized: now 1786069478428505 us; error 48 us; skew 500 ppm
Aug 07 02:24:38 dist-test-slave-69rg krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069478, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:38.431584 20349 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:24:38.432762 20349 webserver.cc:460] Webserver started at http://127.4.221.65:45795/ using document root <none> and password file <none>
I20260807 02:24:38.433370 20349 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:38.433436 20349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:38.437600 20349 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260807 02:24:38.439926 20401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:38.441218 20349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:24:38.441360 20349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "197797ffdacf4f3cb08d28ad7b071103"
format_stamp: "Formatted at 2026-08-07 02:24:35 on dist-test-slave-69rg"
server_key: "7f167da237ac21a3a4fed6b424367627"
server_key_iv: "e47c55ef4712599d955ec02fcb703078"
server_key_version: "encryptionkey@0"
I20260807 02:24:38.441985 20349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:38.472383 20349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:38.475847 20349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:38.476085 20349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:38.476783 20349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:38.477984 20409 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 02:24:38.483479 20349 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 02:24:38.483562 20349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260807 02:24:38.483667 20349 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 02:24:38.485371 20349 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 02:24:38.485446 20349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260807 02:24:38.485726 20409 tablet_bootstrap.cc:492] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103: Bootstrap starting.
I20260807 02:24:38.496883 20349 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:38365
I20260807 02:24:38.497143 20520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:38365 every 8 connection(s)
I20260807 02:24:38.498059 20349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:24:38.503295 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 20349
I20260807 02:24:38.503737 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:33005
--local_ip_for_outbound_sockets=127.4.221.66
--tserver_master_addrs=127.4.221.126:44123
--webserver_port=32851
--webserver_interface=127.4.221.66
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:24:38.507959 20409 log.cc:824] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:38.518437 20409 tablet_bootstrap.cc:492] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:24:38.518922 20409 tablet_bootstrap.cc:492] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103: Bootstrap complete.
I20260807 02:24:38.520558 20409 ts_tablet_manager.cc:1397] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103: Time spent bootstrapping tablet: real 0.035s user 0.022s sys 0.012s
W20260807 02:24:38.521430 20524 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:24:38.500140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:46779)
0807 02:24:38.500657 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:38.500713 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:38.501564 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:38.520411 (+ 18847us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":156,"threads_started":1}
W20260807 02:24:38.522154 20521 heartbeater.cc:644] Failed to heartbeat to 127.4.221.126:44123 (0 consecutive failures): Network error: Failed to ping master at 127.4.221.126:44123: Client connection negotiation failed: client connection to 127.4.221.126:44123: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 02:24:38.523144 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:38.522789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:49543)
0807 02:24:38.522969 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:38.523061 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260807 02:24:38.523923 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:38.523700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:49091)
0807 02:24:38.523811 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:38.523867 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 02:24:38.524130 20521 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 02:24:38.523803 20409 raft_consensus.cc:348] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:38.524382 20409 raft_consensus.cc:729] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 197797ffdacf4f3cb08d28ad7b071103, State: Initialized, Role: FOLLOWER
I20260807 02:24:38.524849 20409 consensus_queue.cc:260] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:38.526007 20409 ts_tablet_manager.cc:1428] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 02:24:38.526044 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:38.525769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:58523)
0807 02:24:38.525931 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:38.525984 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260807 02:24:38.526706 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:38.526508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:54555)
0807 02:24:38.526616 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:38.526660 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260807 02:24:38.624418 20525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:38.624691 20525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:38.624737 20525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:38.628969 20525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:38.629056 20525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:38.629163 20525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:24:38.634934 20525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:33005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=32851
--tserver_master_addrs=127.4.221.126:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:38.636142 20525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:38.637265 20525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:38.648674 20532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:38.648646 20533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:38.648793 20525 server_base.cc:1034] running on GCE node
W20260807 02:24:38.649000 20535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:38.649498 20525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:38.650211 20525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:38.651430 20525 hybrid_clock.cc:648] HybridClock initialized: now 1786069478651416 us; error 28 us; skew 500 ppm
Aug 07 02:24:38 dist-test-slave-69rg krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069478, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:38.654479 20525 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:24:38.655439 20525 webserver.cc:460] Webserver started at http://127.4.221.66:32851/ using document root <none> and password file <none>
I20260807 02:24:38.655982 20525 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:38.656044 20525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:38.660028 20525 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:24:38.662453 20542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:38.664235 20525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:24:38.664354 20525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "631ebf3e199c415abdc52720f90bc90e"
format_stamp: "Formatted at 2026-08-07 02:24:36 on dist-test-slave-69rg"
server_key: "bfd3023565641e90f6fc7bcd21e548c4"
server_key_iv: "4d1c086f9c457fb89870afdb2a0113dd"
server_key_version: "encryptionkey@0"
I20260807 02:24:38.664965 20525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:38.695248 20525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:38.698421 20525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:38.698649 20525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:38.699294 20525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:38.700474 20549 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 02:24:38.704923 20525 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 02:24:38.705008 20525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260807 02:24:38.705106 20525 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 02:24:38.706979 20525 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 02:24:38.707032 20525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260807 02:24:38.707376 20549 tablet_bootstrap.cc:492] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e: Bootstrap starting.
I20260807 02:24:38.712966 20549 log.cc:824] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:38.718333 20525 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:33005
I20260807 02:24:38.718387 20667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:33005 every 8 connection(s)
I20260807 02:24:38.719218 20525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:24:38.721302 20549 tablet_bootstrap.cc:492] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:24:38.721752 20549 tablet_bootstrap.cc:492] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e: Bootstrap complete.
I20260807 02:24:38.722150 20549 ts_tablet_manager.cc:1397] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20260807 02:24:38.722805 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:38.721132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:55807)
0807 02:24:38.721697 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:38.721876 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":155,"threads_started":1}
W20260807 02:24:38.723264 20668 heartbeater.cc:644] Failed to heartbeat to 127.4.221.126:44123 (0 consecutive failures): Network error: Failed to ping master at 127.4.221.126:44123: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
I20260807 02:24:38.723956 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 20525
I20260807 02:24:38.724023 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:38.723717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:33179)
0807 02:24:38.723870 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:38.723931 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260807 02:24:38.724473 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:41487
--local_ip_for_outbound_sockets=127.4.221.67
--tserver_master_addrs=127.4.221.126:44123
--webserver_port=44153
--webserver_interface=127.4.221.67
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:40183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:24:38.724841 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:38.724577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:58187)
0807 02:24:38.724720 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:38.724778 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260807 02:24:38.725037 20668 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 02:24:38.726001 20549 raft_consensus.cc:348] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:38.726446 20549 raft_consensus.cc:729] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 631ebf3e199c415abdc52720f90bc90e, State: Initialized, Role: FOLLOWER
I20260807 02:24:38.726893 20549 consensus_queue.cc:260] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:38.727857 20549 ts_tablet_manager.cc:1428] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 02:24:38.728092 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:38.727844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:55051)
0807 02:24:38.727975 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:38.728029 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 02:24:38.728890 20670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:38.728609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:43257)
0807 02:24:38.728751 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:38.728812 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 02:24:38.840194 20671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:38.840473 20671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:38.840517 20671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:38.844872 20671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:38.844939 20671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:38.845029 20671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:24:38.850589 20671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:40183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:41487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=44153
--tserver_master_addrs=127.4.221.126:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:38.851701 20671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:38.852730 20671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:38.862649 20681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:38.862599 20678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:38.862598 20679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:38.862951 20671 server_base.cc:1034] running on GCE node
I20260807 02:24:38.863372 20671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:38.864080 20671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:38.865276 20671 hybrid_clock.cc:648] HybridClock initialized: now 1786069478865254 us; error 42 us; skew 500 ppm
Aug 07 02:24:38 dist-test-slave-69rg krb5kdc[19138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069478, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:38.868322 20671 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:24:38.869366 20671 webserver.cc:460] Webserver started at http://127.4.221.67:44153/ using document root <none> and password file <none>
I20260807 02:24:38.870011 20671 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:38.870107 20671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:38.874207 20671 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260807 02:24:38.876515 20688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:38.878008 20671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:24:38.878170 20671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "f34d7c5265af4d91b341795bcf5f9f1a"
format_stamp: "Formatted at 2026-08-07 02:24:36 on dist-test-slave-69rg"
server_key: "85a138e2ed3a42ba1f058677ccbd56f8"
server_key_iv: "bb9635437cf0e524c607be9f9cd9c66e"
server_key_version: "encryptionkey@0"
I20260807 02:24:38.878777 20671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:38.905434 20671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:38.908923 20671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:38.909158 20671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:38.909801 20671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:38.910739 20671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:24:38.910813 20671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:38.910890 20671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:24:38.910933 20671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:38.920368 20671 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:41487
I20260807 02:24:38.920408 20803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:41487 every 8 connection(s)
I20260807 02:24:38.921471 20671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:24:38.923815 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 20671
I20260807 02:24:38.925626 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:38.923741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:39977)
0807 02:24:38.924260 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:38.924815 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":168,"threads_started":1}
W20260807 02:24:38.926088 20804 heartbeater.cc:644] Failed to heartbeat to 127.4.221.126:44123 (0 consecutive failures): Network error: Failed to ping master at 127.4.221.126:44123: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
I20260807 02:24:38.926679 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:38.926451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:38397)
0807 02:24:38.926565 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:38.926612 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 02:24:38.927301 20806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:38.927093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:53955)
0807 02:24:38.927198 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:38.927243 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260807 02:24:38.927477 20804 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 02:24:39.528092 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:39.527561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:51593)
0807 02:24:39.527871 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:39.527933 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":106,"threads_started":1}
I20260807 02:24:39.673231 20809 raft_consensus.cc:482] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:24:39.673530 20809 raft_consensus.cc:504] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:39.675083 20809 leader_election.cc:288] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f34d7c5265af4d91b341795bcf5f9f1a (127.4.221.67:41487), 631ebf3e199c415abdc52720f90bc90e (127.4.221.66:33005)
Aug 07 02:24:39 dist-test-slave-69rg krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069478, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.66@KRBTEST.COM
Aug 07 02:24:39 dist-test-slave-69rg krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069478, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.67@KRBTEST.COM
I20260807 02:24:39.693845 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:39.675334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:41487 (local address 127.4.221.65:42397)
0807 02:24:39.675566 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:39.675583 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:39.675719 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:39.677793 (+ 2074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:39.677796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:39.678277 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:39.679352 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:39.679378 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:39.681422 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:39.681426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:39.683240 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:39.683251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:39.684381 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:39.684385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:39.684745 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:39.685683 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:39.685707 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:39.688562 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:39.690814 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:39.690821 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:39.690838 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:39.691221 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:39.691648 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:39.691652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:39.691654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:39.691819 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:39.692813 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:39.692818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:39.693053 (+ 235us) client_negotiation.cc:769] Sending connection context
0807 02:24:39.693209 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 02:24:39.693596 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":477,"spinlock_wait_cycles":39808}
I20260807 02:24:39.694024 20811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:39.675566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:42397 (local address 127.4.221.67:41487)
0807 02:24:39.676117 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:39.676129 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:39.676163 (+ 34us) server_negotiation.cc:407] Connection header received
0807 02:24:39.677129 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:39.677137 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:39.677533 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:39.677848 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:39.679568 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:39.681233 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:39.683441 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:39.684226 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:39.684451 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:39.688772 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:39.688776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:39.688786 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:39.688812 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:39.690688 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:39.691372 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:39.691416 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:39.691420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:39.691540 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:39.691951 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:39.691955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:39.691957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:39.692694 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:39.692854 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:39.693420 (+ 566us) server_negotiation.cc:299] Negotiation successful
0807 02:24:39.693775 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":136,"threads_started":1}
I20260807 02:24:39.693864 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:39.675511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:33005 (local address 127.4.221.65:52301)
0807 02:24:39.675927 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:39.675944 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:39.676069 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:39.677528 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:39.677534 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:39.678279 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:39.679352 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:39.679385 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:39.681439 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:39.681444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:39.683243 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:39.683251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:39.684381 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:39.684394 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:39.684748 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:39.685692 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:39.685712 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:39.688026 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:39.691839 (+ 3813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:39.691845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:39.691850 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:39.692071 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:39.692525 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:39.692528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:39.692529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:39.692607 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:39.693330 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:39.693333 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:39.693492 (+ 159us) client_negotiation.cc:769] Sending connection context
0807 02:24:39.693595 (+ 103us) client_negotiation.cc:240] Negotiation successful
0807 02:24:39.693714 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":562,"spinlock_wait_cycles":6656,"thread_start_us":82,"threads_started":1}
I20260807 02:24:39.694358 20812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:39.675668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:52301 (local address 127.4.221.66:33005)
0807 02:24:39.676053 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:39.676073 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:39.676128 (+ 55us) server_negotiation.cc:407] Connection header received
0807 02:24:39.676485 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:39.676492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:39.677051 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:39.677450 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:39.679568 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:39.681234 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:39.683426 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:39.684226 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:39.684430 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:39.688287 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:39.688294 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:39.688313 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:39.688348 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:39.691722 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:39.692251 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:39.692257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:39.692263 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:39.692415 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:39.692719 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:39.692723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:39.692726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:39.693212 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:39.693353 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:39.693775 (+ 422us) server_negotiation.cc:299] Negotiation successful
0807 02:24:39.694154 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":80,"threads_started":1}
I20260807 02:24:39.695605 20619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3dda852e2b4bec96b00622ecd7bf14" candidate_uuid: "197797ffdacf4f3cb08d28ad7b071103" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "631ebf3e199c415abdc52720f90bc90e" is_pre_election: true
I20260807 02:24:39.696014 20619 raft_consensus.cc:2454] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 197797ffdacf4f3cb08d28ad7b071103 in term 1.
I20260807 02:24:39.697235 20406 leader_election.cc:302] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 197797ffdacf4f3cb08d28ad7b071103, 631ebf3e199c415abdc52720f90bc90e; no voters:
I20260807 02:24:39.697650 20809 raft_consensus.cc:2781] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260807 02:24:39.697772 20809 raft_consensus.cc:482] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:24:39.697863 20809 raft_consensus.cc:3037] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 1 FOLLOWER]: Advancing to term 2
W20260807 02:24:39.698294 20402 leader_election.cc:341] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f34d7c5265af4d91b341795bcf5f9f1a (127.4.221.67:41487): Not found: Tablet not found: 8c3dda852e2b4bec96b00622ecd7bf14
I20260807 02:24:39.700615 20809 raft_consensus.cc:504] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:39.701071 20809 leader_election.cc:288] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [CANDIDATE]: Term 2 election: Requested vote from peers f34d7c5265af4d91b341795bcf5f9f1a (127.4.221.67:41487), 631ebf3e199c415abdc52720f90bc90e (127.4.221.66:33005)
I20260807 02:24:39.701617 20619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3dda852e2b4bec96b00622ecd7bf14" candidate_uuid: "197797ffdacf4f3cb08d28ad7b071103" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "631ebf3e199c415abdc52720f90bc90e"
I20260807 02:24:39.701803 20619 raft_consensus.cc:3037] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 1 FOLLOWER]: Advancing to term 2
W20260807 02:24:39.701994 20402 leader_election.cc:341] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer f34d7c5265af4d91b341795bcf5f9f1a (127.4.221.67:41487): Not found: Tablet not found: 8c3dda852e2b4bec96b00622ecd7bf14
I20260807 02:24:39.704659 20619 raft_consensus.cc:2454] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 197797ffdacf4f3cb08d28ad7b071103 in term 2.
I20260807 02:24:39.705041 20406 leader_election.cc:302] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 197797ffdacf4f3cb08d28ad7b071103, 631ebf3e199c415abdc52720f90bc90e; no voters: f34d7c5265af4d91b341795bcf5f9f1a
I20260807 02:24:39.705260 20809 raft_consensus.cc:2781] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 02:24:39.705492 20809 raft_consensus.cc:686] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [term 2 LEADER]: Becoming Leader. State: Replica: 197797ffdacf4f3cb08d28ad7b071103, State: Running, Role: LEADER
I20260807 02:24:39.705997 20809 consensus_queue.cc:237] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:39.706396 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:39.706106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:39673)
0807 02:24:39.706246 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:39.706312 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260807 02:24:39.730278 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:39.729723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:42505)
0807 02:24:39.730101 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:39.730170 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":172,"threads_started":1}
I20260807 02:24:39.928642 20816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:39.928114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:40623)
0807 02:24:39.928457 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:39.928524 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":125,"threads_started":1}
I20260807 02:24:40.105283 20619 raft_consensus.cc:1261] T 8c3dda852e2b4bec96b00622ecd7bf14 P 631ebf3e199c415abdc52720f90bc90e [term 2 FOLLOWER]: Refusing update from remote peer 197797ffdacf4f3cb08d28ad7b071103: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260807 02:24:40.106292 20809 consensus_queue.cc:1035] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 [LEADER]: Connected to new peer: Peer: permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260807 02:24:40.109009 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:40.108642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:52281)
0807 02:24:40.108808 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:40.108908 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260807 02:24:40.112274 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:40.111901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:49027)
0807 02:24:40.112098 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:40.112165 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260807 02:24:40.112423 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:40.112133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:41777)
0807 02:24:40.112257 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:40.112322 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 02:24:40.112576 20402 consensus_peers.cc:487] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103 -> Peer f34d7c5265af4d91b341795bcf5f9f1a (127.4.221.67:41487): Couldn't send request to peer f34d7c5265af4d91b341795bcf5f9f1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c3dda852e2b4bec96b00622ecd7bf14. This is attempt 1: this message will repeat every 5th retry.
I20260807 02:24:40.543308 20822 ts_tablet_manager.cc:927] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: Initiating tablet copy from peer 197797ffdacf4f3cb08d28ad7b071103 (127.4.221.65:38365)
I20260807 02:24:40.544171 20822 tablet_copy_client.cc:316] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: tablet copy: Beginning tablet copy session from remote peer at address 127.4.221.65:38365
Aug 07 02:24:40 dist-test-slave-69rg krb5kdc[19138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069478, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.65@KRBTEST.COM
I20260807 02:24:40.558532 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:40.545012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:38365 (local address 127.4.221.67:54945)
0807 02:24:40.545576 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:40.545600 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:40.545787 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:40.546553 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:40.546579 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:40.547115 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:40.547266 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:40.547274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:40.548545 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:40.548549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:40.549450 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:40.549464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:40.550522 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:40.550526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:40.550645 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:40.551524 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:40.551557 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:40.553526 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:40.556166 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:40.556181 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:40.556190 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:40.556622 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:40.556999 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:40.557002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:40.557003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:40.557097 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:40.557784 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:40.557791 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:40.557994 (+ 203us) client_negotiation.cc:769] Sending connection context
0807 02:24:40.558157 (+ 163us) client_negotiation.cc:240] Negotiation successful
0807 02:24:40.558356 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":151,"threads_started":1}
I20260807 02:24:40.558676 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:40.545194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:54945 (local address 127.4.221.65:38365)
0807 02:24:40.545528 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:40.545540 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:40.545745 (+ 205us) server_negotiation.cc:407] Connection header received
0807 02:24:40.545983 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:40.545990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:40.546318 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:40.546523 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:40.547459 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:40.548383 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:40.549602 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:40.550347 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:40.550452 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:40.553770 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:40.553786 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:40.553805 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:40.553904 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:40.556020 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:40.556773 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:40.556780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:40.556782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:40.556891 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:40.557216 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:40.557226 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:40.557227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:40.557665 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:40.557806 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:40.558276 (+ 470us) server_negotiation.cc:299] Negotiation successful
0807 02:24:40.558486 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":73,"threads_started":1}
I20260807 02:24:40.560012 20494 tablet_copy_service.cc:140] P 197797ffdacf4f3cb08d28ad7b071103: Received BeginTabletCopySession request for tablet 8c3dda852e2b4bec96b00622ecd7bf14 from peer f34d7c5265af4d91b341795bcf5f9f1a ({username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:54945)
I20260807 02:24:40.560207 20494 tablet_copy_service.cc:161] P 197797ffdacf4f3cb08d28ad7b071103: Beginning new tablet copy session on tablet 8c3dda852e2b4bec96b00622ecd7bf14 from peer f34d7c5265af4d91b341795bcf5f9f1a at {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:54945: session id = f34d7c5265af4d91b341795bcf5f9f1a-8c3dda852e2b4bec96b00622ecd7bf14
I20260807 02:24:40.561683 20494 tablet_copy_source_session.cc:205] T 8c3dda852e2b4bec96b00622ecd7bf14 P 197797ffdacf4f3cb08d28ad7b071103: Tablet Copy: opened 0 blocks and 1 log segments
I20260807 02:24:40.566990 20822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3dda852e2b4bec96b00622ecd7bf14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:40.575805 20822 tablet_copy_client.cc:791] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: tablet copy: Starting download of 0 data blocks...
I20260807 02:24:40.576085 20822 tablet_copy_client.cc:655] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: tablet copy: Starting download of 1 WAL segments...
I20260807 02:24:40.578459 20822 tablet_copy_client.cc:531] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260807 02:24:40.580834 20822 tablet_bootstrap.cc:492] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: Bootstrap starting.
I20260807 02:24:40.587847 20822 log.cc:824] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:40.595888 20822 tablet_bootstrap.cc:492] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:24:40.596246 20822 tablet_bootstrap.cc:492] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: Bootstrap complete.
I20260807 02:24:40.596624 20822 ts_tablet_manager.cc:1397] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: Time spent bootstrapping tablet: real 0.016s user 0.015s sys 0.000s
I20260807 02:24:40.599442 20822 raft_consensus.cc:348] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:40.599745 20822 raft_consensus.cc:729] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f34d7c5265af4d91b341795bcf5f9f1a, State: Initialized, Role: FOLLOWER
I20260807 02:24:40.600255 20822 consensus_queue.cc:260] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f34d7c5265af4d91b341795bcf5f9f1a" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41487 } } peers { permanent_uuid: "631ebf3e199c415abdc52720f90bc90e" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33005 } } peers { permanent_uuid: "197797ffdacf4f3cb08d28ad7b071103" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38365 } }
I20260807 02:24:40.601534 20822 ts_tablet_manager.cc:1428] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260807 02:24:40.601663 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:40.601331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:37595)
0807 02:24:40.601482 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:40.601534 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260807 02:24:40.602427 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:40.602167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:45471)
0807 02:24:40.602277 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:40.602350 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260807 02:24:40.602442 20494 tablet_copy_service.cc:342] P 197797ffdacf4f3cb08d28ad7b071103: Request end of tablet copy session f34d7c5265af4d91b341795bcf5f9f1a-8c3dda852e2b4bec96b00622ecd7bf14 received from {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:54945
I20260807 02:24:40.602581 20494 tablet_copy_service.cc:433] P 197797ffdacf4f3cb08d28ad7b071103: ending tablet copy session f34d7c5265af4d91b341795bcf5f9f1a-8c3dda852e2b4bec96b00622ecd7bf14 on tablet 8c3dda852e2b4bec96b00622ecd7bf14 with peer f34d7c5265af4d91b341795bcf5f9f1a
I20260807 02:24:41.041949 20830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:41.041318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:49433)
0807 02:24:41.041667 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:41.041766 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":71,"threads_started":1}
I20260807 02:24:41.042027 20758 raft_consensus.cc:1203] T 8c3dda852e2b4bec96b00622ecd7bf14 P f34d7c5265af4d91b341795bcf5f9f1a [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260807 02:24:41.043385 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:41.043037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:60353)
0807 02:24:41.043209 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:41.043283 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260807 02:24:41.114039 20831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:41.113451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:43007)
0807 02:24:41.113737 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:41.113898 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260807 02:24:42.043594 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:42.043008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:52655)
0807 02:24:42.043361 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:42.043462 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":120,"threads_started":1}
I20260807 02:24:42.044759 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:42.044274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:41579)
0807 02:24:42.044568 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:42.044631 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":114,"threads_started":1}
I20260807 02:24:42.115805 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:42.115135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:45047)
0807 02:24:42.115575 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:42.115668 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":108,"threads_started":1}
I20260807 02:24:43.045526 20836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:43.044798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:33985)
0807 02:24:43.045240 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:43.045339 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":189,"threads_started":1}
I20260807 02:24:43.046550 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:43.045798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:52461)
0807 02:24:43.046312 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:43.046405 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":98,"threads_started":1}
I20260807 02:24:43.117480 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:43.116933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:43923)
0807 02:24:43.117270 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:43.117342 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":106,"threads_started":1}
I20260807 02:24:44.047515 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:44.046938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:56343)
0807 02:24:44.047296 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:44.047382 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":152,"threads_started":1}
I20260807 02:24:44.048178 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:44.047630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:35207)
0807 02:24:44.047992 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:44.048078 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":93,"threads_started":1}
I20260807 02:24:44.048458 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:44.048180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:58809)
0807 02:24:44.048327 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:44.048388 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260807 02:24:44.119371 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:44.118821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:42207)
0807 02:24:44.119159 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:44.119232 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20260807 02:24:45.049991 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:45.049314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:57641)
0807 02:24:45.049733 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:45.049797 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":200,"threads_started":1}
I20260807 02:24:45.049989 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:45.049340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:44075)
0807 02:24:45.049727 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:45.049794 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":170,"threads_started":1}
I20260807 02:24:45.120985 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:45.120438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:54827)
0807 02:24:45.120780 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:45.120850 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":100,"threads_started":1}
I20260807 02:24:46.051756 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:46.051187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:48777)
0807 02:24:46.051573 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:46.051640 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":136,"threads_started":1}
I20260807 02:24:46.051756 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:46.051205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:50691)
0807 02:24:46.051544 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:46.051624 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":119,"threads_started":1}
I20260807 02:24:46.122567 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:46.121948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:47863)
0807 02:24:46.122317 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:46.122384 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":124,"threads_started":1}
I20260807 02:24:47.053376 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:47.052783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:49967)
0807 02:24:47.053175 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:47.053244 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":146,"threads_started":1}
I20260807 02:24:47.053376 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:47.052780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:54461)
0807 02:24:47.053161 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:47.053244 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":108,"threads_started":1}
I20260807 02:24:47.124455 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:47.123725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:60637)
0807 02:24:47.124199 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:47.124283 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":196,"threads_started":1}
I20260807 02:24:48.055249 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:48.054578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:40997)
0807 02:24:48.055000 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:48.055087 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":121,"threads_started":1}
I20260807 02:24:48.055249 20851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:48.054588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:60023)
0807 02:24:48.055064 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:48.055130 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":136,"threads_started":1}
I20260807 02:24:48.126274 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:48.125646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:36945)
0807 02:24:48.126067 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:48.126142 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":105,"threads_started":1}
I20260807 02:24:49.057292 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:49.056572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:34781)
0807 02:24:49.057055 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:49.057155 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":123,"threads_started":1}
I20260807 02:24:49.057288 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:49.056571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:46577)
0807 02:24:49.057076 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:49.057150 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":160,"threads_started":1}
I20260807 02:24:49.128093 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:49.127494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:37275)
0807 02:24:49.127859 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:49.127931 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":135,"threads_started":1}
I20260807 02:24:50.059144 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:50.058508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:51027)
0807 02:24:50.058926 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:50.058985 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":132,"threads_started":1}
I20260807 02:24:50.059124 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:50.058483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:41075)
0807 02:24:50.058877 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:50.058962 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":140,"threads_started":1}
I20260807 02:24:50.130017 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:50.129313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:45039)
0807 02:24:50.129711 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:50.129796 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":144,"threads_started":1}
I20260807 02:24:51.060817 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:51.060250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:42175)
0807 02:24:51.060607 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:51.060682 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":132,"threads_started":1}
I20260807 02:24:51.060905 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:51.060250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:50641)
0807 02:24:51.060730 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:51.060799 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":133,"threads_started":1}
I20260807 02:24:51.131811 20863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:51.131146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:54347)
0807 02:24:51.131608 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:51.131680 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":236,"threads_started":1}
I20260807 02:24:52.062675 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:52.061996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:33981)
0807 02:24:52.062481 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:52.062545 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":151,"threads_started":1}
I20260807 02:24:52.062675 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:52.061991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:55081)
0807 02:24:52.062384 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:52.062518 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":145,"threads_started":1}
I20260807 02:24:52.133611 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:52.132942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:46067)
0807 02:24:52.133363 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:52.133469 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":154,"threads_started":1}
I20260807 02:24:53.064417 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:53.063811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.65:37707)
0807 02:24:53.064234 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:53.064300 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":140,"threads_started":1}
I20260807 02:24:53.064416 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:53.063762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.67:47625)
0807 02:24:53.064219 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:53.064286 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":125,"threads_started":1}
I20260807 02:24:53.135329 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:53.134749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44123 (local address 127.4.221.66:37573)
0807 02:24:53.135119 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:53.135192 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":120,"threads_started":1}
I20260807 02:24:53.934450 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 20349
I20260807 02:24:53.943671 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 20525
I20260807 02:24:53.952847 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 20671
2026-08-07T02:24:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19717 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:24:54 dist-test-slave-69rg krb5kdc[20873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:24:54 dist-test-slave-69rg krb5kdc[20873](info): set up 2 sockets
Aug 07 02:24:54 dist-test-slave-69rg krb5kdc[20873](info): commencing operation
krb5kdc: starting...
W20260807 02:24:55.025784 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:24:55 dist-test-slave-69rg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:24:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:24:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:24:55.161912 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:45977
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:45977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:55.278432 20887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:55.278744 20887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:55.278836 20887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:55.282953 20887 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 02:24:55.283036 20887 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 02:24:55.283064 20887 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 02:24:55.283102 20887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:24:55.283143 20887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:55.283183 20887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:55.283217 20887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:24:55.283248 20887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:24:55.288704 20887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:45977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:45977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:55.289915 20887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:55.290972 20887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:55.300482 20896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:55.300489 20894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:55.300534 20892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:55.300547 20887 server_base.cc:1034] running on GCE node
I20260807 02:24:55.301426 20887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:55.302652 20887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:55.303853 20887 hybrid_clock.cc:648] HybridClock initialized: now 1786069495303836 us; error 34 us; skew 500 ppm
Aug 07 02:24:55 dist-test-slave-69rg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069495, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:55.306840 20887 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:24:55.307773 20887 webserver.cc:460] Webserver started at http://127.4.221.126:43253/ using document root <none> and password file <none>
I20260807 02:24:55.308310 20887 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:55.308372 20887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:55.308542 20887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:24:55.310691 20887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "f56a6067d16f482a90c0d0c5f75382f6"
format_stamp: "Formatted at 2026-08-07 02:24:55 on dist-test-slave-69rg"
server_key: "93e2e995ff605092a9af472331b22272"
server_key_iv: "484c7ef14169b8983b753471dffbc885"
server_key_version: "encryptionkey@0"
I20260807 02:24:55.311229 20887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "f56a6067d16f482a90c0d0c5f75382f6"
format_stamp: "Formatted at 2026-08-07 02:24:55 on dist-test-slave-69rg"
server_key: "93e2e995ff605092a9af472331b22272"
server_key_iv: "484c7ef14169b8983b753471dffbc885"
server_key_version: "encryptionkey@0"
I20260807 02:24:55.314946 20887 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:24:55.317365 20903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:55.318653 20887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:24:55.318759 20887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "f56a6067d16f482a90c0d0c5f75382f6"
format_stamp: "Formatted at 2026-08-07 02:24:55 on dist-test-slave-69rg"
server_key: "93e2e995ff605092a9af472331b22272"
server_key_iv: "484c7ef14169b8983b753471dffbc885"
server_key_version: "encryptionkey@0"
I20260807 02:24:55.318861 20887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:55.361972 20887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:55.365298 20887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:55.365469 20887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:55.372855 20887 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:45977
I20260807 02:24:55.373163 20958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:45977 every 8 connection(s)
I20260807 02:24:55.373917 20887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:24:55.376904 20959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:55.380671 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 20887
I20260807 02:24:55.380820 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:24:55.381147 4981 external_mini_cluster.cc:1428] Setting key b9c8c3bfd54a7ab883856d091b980858
I20260807 02:24:55.383221 20959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Bootstrap starting.
I20260807 02:24:55.385910 20959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:55.386751 20959 log.cc:824] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:55.389111 20959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: No bootstrap required, opened a new log
Aug 07 02:24:55 dist-test-slave-69rg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:55.393448 20959 raft_consensus.cc:348] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } }
I20260807 02:24:55.393692 20959 raft_consensus.cc:374] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:55.393781 20959 raft_consensus.cc:729] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56a6067d16f482a90c0d0c5f75382f6, State: Initialized, Role: FOLLOWER
I20260807 02:24:55.394340 20959 consensus_queue.cc:260] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } }
I20260807 02:24:55.394520 20959 raft_consensus.cc:388] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:24:55.394604 20959 raft_consensus.cc:482] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:24:55.394760 20959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:55.395917 20959 raft_consensus.cc:504] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } }
I20260807 02:24:55.396298 20959 leader_election.cc:302] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f56a6067d16f482a90c0d0c5f75382f6; no voters:
I20260807 02:24:55.396351 20962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:55.382724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47910 (local address 127.4.221.126:45977)
0807 02:24:55.383193 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:55.383209 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:55.383246 (+ 37us) server_negotiation.cc:407] Connection header received
0807 02:24:55.383961 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:55.383979 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:55.384368 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:55.384665 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:55.385586 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:55.386724 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:55.387787 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:55.388252 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:55.388403 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:55.391287 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:55.391292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:55.391307 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:55.391342 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:55.393118 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:55.393594 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:55.393600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:55.393605 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:55.393784 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:55.394163 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:55.394166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:55.394168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:55.394650 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:55.394892 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:55.395163 (+ 271us) server_negotiation.cc:299] Negotiation successful
0807 02:24:55.395495 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":147,"threads_started":1}
I20260807 02:24:55.396637 20959 leader_election.cc:288] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:24:55.396842 20964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:24:55.397094 20964 raft_consensus.cc:686] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 1 LEADER]: Becoming Leader. State: Replica: f56a6067d16f482a90c0d0c5f75382f6, State: Running, Role: LEADER
I20260807 02:24:55.397476 20964 consensus_queue.cc:237] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } }
I20260807 02:24:55.398098 20959 sys_catalog.cc:564] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:24:55.399385 20966 sys_catalog.cc:455] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f56a6067d16f482a90c0d0c5f75382f6. Latest consensus state: current_term: 1 leader_uuid: "f56a6067d16f482a90c0d0c5f75382f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } } }
I20260807 02:24:55.399345 20965 sys_catalog.cc:455] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f56a6067d16f482a90c0d0c5f75382f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } } }
I20260807 02:24:55.399499 20966 sys_catalog.cc:458] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:55.399499 20965 sys_catalog.cc:458] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [sys.catalog]: This master's current role is: LEADER
I20260807 02:24:55.399945 20974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:24:55.403342 20974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:24:55.409284 20974 catalog_manager.cc:1347] Generated new cluster ID: d385d383e608403193d5e65db8fee151
I20260807 02:24:55.409425 20974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:24:55.419660 20974 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:24:55.420965 20974 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:24:55.428717 20974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Generated new TSK 0
I20260807 02:24:55.429590 20974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:24:55.473578 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:45977
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:55.591935 20987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:55.592226 20987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:55.592272 20987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:55.596372 20987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:55.596442 20987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:55.596503 20987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:24:55.602334 20987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:55.603437 20987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:55.604451 20987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:55.614562 20993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:55.614552 20996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:55.614552 20994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:55.614805 20987 server_base.cc:1034] running on GCE node
I20260807 02:24:55.615228 20987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:55.615944 20987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:55.617182 20987 hybrid_clock.cc:648] HybridClock initialized: now 1786069495617149 us; error 46 us; skew 500 ppm
Aug 07 02:24:55 dist-test-slave-69rg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069495, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:55.620344 20987 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:24:55.621286 20987 webserver.cc:460] Webserver started at http://127.4.221.65:34421/ using document root <none> and password file <none>
I20260807 02:24:55.621986 20987 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:55.622089 20987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:55.622326 20987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:24:55.624503 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "8c5489a1ae504bc5b0466e9652c1cb0a"
format_stamp: "Formatted at 2026-08-07 02:24:55 on dist-test-slave-69rg"
server_key: "cca6f77050a51fd3a77688a33d438748"
server_key_iv: "8536e69d0f6a730b4c95ad29b574b459"
server_key_version: "encryptionkey@0"
I20260807 02:24:55.625123 20987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "8c5489a1ae504bc5b0466e9652c1cb0a"
format_stamp: "Formatted at 2026-08-07 02:24:55 on dist-test-slave-69rg"
server_key: "cca6f77050a51fd3a77688a33d438748"
server_key_iv: "8536e69d0f6a730b4c95ad29b574b459"
server_key_version: "encryptionkey@0"
I20260807 02:24:55.629077 20987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260807 02:24:55.631831 21003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:55.633091 20987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:24:55.633239 20987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "8c5489a1ae504bc5b0466e9652c1cb0a"
format_stamp: "Formatted at 2026-08-07 02:24:55 on dist-test-slave-69rg"
server_key: "cca6f77050a51fd3a77688a33d438748"
server_key_iv: "8536e69d0f6a730b4c95ad29b574b459"
server_key_version: "encryptionkey@0"
I20260807 02:24:55.633383 20987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:55.657225 20987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:55.660708 20987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:55.660955 20987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:55.661656 20987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:55.662550 20987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:24:55.662627 20987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:55.662711 20987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:24:55.662766 20987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:55.672222 20987 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:44017
I20260807 02:24:55.672253 21124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:44017 every 8 connection(s)
I20260807 02:24:55.673154 20987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:24:55.682166 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 20987
I20260807 02:24:55.682286 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:24:55.682611 4981 external_mini_cluster.cc:1428] Setting key e68cdd5a7a8f35f98d5ca2891769ad62
Aug 07 02:24:55 dist-test-slave-69rg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069495, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:55.688094 20962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:55.675146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:34581 (local address 127.4.221.126:45977)
0807 02:24:55.675317 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:55.675321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:55.676181 (+ 860us) server_negotiation.cc:407] Connection header received
0807 02:24:55.677533 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:55.677537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:55.677597 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:55.677707 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:55.679000 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:55.679559 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:55.680747 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:55.681006 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:55.681118 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:55.684009 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:55.684013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:55.684015 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:55.684045 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:55.685596 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:55.686346 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:55.686353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:55.686357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:55.686466 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:55.686851 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:55.686856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:55.686858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:55.687127 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:55.687240 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:55.687805 (+ 565us) server_negotiation.cc:299] Negotiation successful
0807 02:24:55.687967 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 02:24:55.689260 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:55.675551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:34581)
0807 02:24:55.676019 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:55.676065 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:55.677310 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:55.677915 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:55.677922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:55.678315 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:55.678831 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:55.678851 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:55.679695 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:55.679699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:55.680603 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:55.680613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:55.681179 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:55.681183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:55.681371 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:55.682060 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:55.682077 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:55.683785 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:55.685734 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:55.685743 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:55.685763 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:55.686197 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:55.686582 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:55.686584 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:55.686586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:55.686734 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:55.687267 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:55.687272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:55.687534 (+ 262us) client_negotiation.cc:769] Sending connection context
0807 02:24:55.687691 (+ 157us) client_negotiation.cc:240] Negotiation successful
0807 02:24:55.688090 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":119,"threads_started":1}
I20260807 02:24:55.690675 21125 heartbeater.cc:347] Connected to a master server at 127.4.221.126:45977
I20260807 02:24:55.690970 21125 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:55.691656 21125 heartbeater.cc:511] Master 127.4.221.126:45977 requested a full tablet report, sending...
I20260807 02:24:55.693291 20920 ts_manager.cc:194] Registered new tserver with Master: 8c5489a1ae504bc5b0466e9652c1cb0a (127.4.221.65:44017)
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
I20260807 02:24:55.694604 20920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:34581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:24:55.716653 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:45977
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:55.833758 21130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:55.834170 21130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:55.834259 21130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:55.838593 21130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:55.838696 21130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:55.838791 21130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:24:55.844283 21130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:55.845564 21130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:55.846717 21130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:55.856532 21137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:55.856638 21139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:55.856649 21135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:55.856984 21130 server_base.cc:1034] running on GCE node
I20260807 02:24:55.857483 21130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:55.858222 21130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:55.859419 21130 hybrid_clock.cc:648] HybridClock initialized: now 1786069495859404 us; error 29 us; skew 500 ppm
Aug 07 02:24:55 dist-test-slave-69rg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069495, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:55.862341 21130 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:24:55.863322 21130 webserver.cc:460] Webserver started at http://127.4.221.66:37791/ using document root <none> and password file <none>
I20260807 02:24:55.863961 21130 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:55.864059 21130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:55.864305 21130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:24:55.866669 21130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "db28e04bd78c460284a5f1561c6bb912"
format_stamp: "Formatted at 2026-08-07 02:24:55 on dist-test-slave-69rg"
server_key: "f0da3b3baebbeaf8963a1948416145bd"
server_key_iv: "c44208ff5c2649179752e03534af1a4b"
server_key_version: "encryptionkey@0"
I20260807 02:24:55.867278 21130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "db28e04bd78c460284a5f1561c6bb912"
format_stamp: "Formatted at 2026-08-07 02:24:55 on dist-test-slave-69rg"
server_key: "f0da3b3baebbeaf8963a1948416145bd"
server_key_iv: "c44208ff5c2649179752e03534af1a4b"
server_key_version: "encryptionkey@0"
I20260807 02:24:55.871335 21130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:24:55.874123 21146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:55.875581 21130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260807 02:24:55.875759 21130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "db28e04bd78c460284a5f1561c6bb912"
format_stamp: "Formatted at 2026-08-07 02:24:55 on dist-test-slave-69rg"
server_key: "f0da3b3baebbeaf8963a1948416145bd"
server_key_iv: "c44208ff5c2649179752e03534af1a4b"
server_key_version: "encryptionkey@0"
I20260807 02:24:55.875900 21130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:55.918390 21130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:55.921617 21130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:55.921907 21130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:55.922544 21130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:55.923491 21130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:24:55.923563 21130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:55.923627 21130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:24:55.923657 21130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:55.932884 21130 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:39553
I20260807 02:24:55.932904 21271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:39553 every 8 connection(s)
I20260807 02:24:55.933805 21130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:24:55.935151 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 21130
I20260807 02:24:55.935279 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:24:55.935581 4981 external_mini_cluster.cc:1428] Setting key daf011118491c0d2bc1033626b4b6f97
Aug 07 02:24:55 dist-test-slave-69rg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069495, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
I20260807 02:24:55.950928 20962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:55.935953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:39723 (local address 127.4.221.126:45977)
0807 02:24:55.936116 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:55.936123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:55.937222 (+ 1099us) server_negotiation.cc:407] Connection header received
0807 02:24:55.938757 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:55.938763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:55.938830 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:55.938929 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:55.940604 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:55.941146 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:55.943058 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:55.943313 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:55.943402 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:55.946507 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:55.946511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:55.946513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:55.946539 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:55.948260 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:55.948848 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:55.948851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:55.948853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:55.948946 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:55.949447 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:55.949450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:55.949452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:55.949668 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:55.949854 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:55.950519 (+ 665us) server_negotiation.cc:299] Negotiation successful
0807 02:24:55.950677 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260807 02:24:55.951727 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:55.936474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:39723)
0807 02:24:55.937050 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:55.937107 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:55.938539 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:55.939086 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:55.939093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:55.939681 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:55.940430 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:55.940451 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:55.941294 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:55.941300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:55.942914 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:55.942930 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:55.943503 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:55.943509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:55.943766 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:55.944467 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:55.944493 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:55.946310 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:55.948403 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:55.948410 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:55.948429 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:55.948722 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:55.949068 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:55.949071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:55.949073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:55.949308 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:55.949784 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:55.949791 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:55.950180 (+ 389us) client_negotiation.cc:769] Sending connection context
0807 02:24:55.950407 (+ 227us) client_negotiation.cc:240] Negotiation successful
0807 02:24:55.950726 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":154,"threads_started":1}
I20260807 02:24:55.953016 21272 heartbeater.cc:347] Connected to a master server at 127.4.221.126:45977
I20260807 02:24:55.953300 21272 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:55.954001 21272 heartbeater.cc:511] Master 127.4.221.126:45977 requested a full tablet report, sending...
I20260807 02:24:55.955156 20920 ts_manager.cc:194] Registered new tserver with Master: db28e04bd78c460284a5f1561c6bb912 (127.4.221.66:39553)
I20260807 02:24:55.956074 20920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:39723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:24:55.970379 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:45977
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:24:56.088299 21277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:24:56.088578 21277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:24:56.088658 21277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:24:56.092732 21277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:24:56.092815 21277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:24:56.092931 21277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:24:56.098574 21277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:24:56.099721 21277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:24:56.100775 21277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:24:56.110857 21284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:56.110857 21283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:24:56.110934 21286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:24:56.111137 21277 server_base.cc:1034] running on GCE node
I20260807 02:24:56.111686 21277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:24:56.112339 21277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:24:56.113531 21277 hybrid_clock.cc:648] HybridClock initialized: now 1786069496113514 us; error 30 us; skew 500 ppm
Aug 07 02:24:56 dist-test-slave-69rg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069496, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:24:56.116377 21277 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:24:56.117254 21277 webserver.cc:460] Webserver started at http://127.4.221.67:40017/ using document root <none> and password file <none>
I20260807 02:24:56.117888 21277 fs_manager.cc:359] Metadata directory not provided
I20260807 02:24:56.117982 21277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:24:56.118238 21277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:24:56.120262 21277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "c316b081e1494e54a9c8b520526e4b37"
format_stamp: "Formatted at 2026-08-07 02:24:56 on dist-test-slave-69rg"
server_key: "e28b8dda8bf7caa0f89352e62f3734aa"
server_key_iv: "0c8487685afa346a61745d6affa13ff4"
server_key_version: "encryptionkey@0"
I20260807 02:24:56.120839 21277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "c316b081e1494e54a9c8b520526e4b37"
format_stamp: "Formatted at 2026-08-07 02:24:56 on dist-test-slave-69rg"
server_key: "e28b8dda8bf7caa0f89352e62f3734aa"
server_key_iv: "0c8487685afa346a61745d6affa13ff4"
server_key_version: "encryptionkey@0"
I20260807 02:24:56.124567 21277 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260807 02:24:56.127051 21293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:56.128350 21277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 02:24:56.128491 21277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "c316b081e1494e54a9c8b520526e4b37"
format_stamp: "Formatted at 2026-08-07 02:24:56 on dist-test-slave-69rg"
server_key: "e28b8dda8bf7caa0f89352e62f3734aa"
server_key_iv: "0c8487685afa346a61745d6affa13ff4"
server_key_version: "encryptionkey@0"
I20260807 02:24:56.128612 21277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:24:56.141209 21277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:24:56.144124 21277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:24:56.144344 21277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:24:56.144958 21277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:24:56.145809 21277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:24:56.145946 21277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:56.146019 21277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:24:56.146062 21277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:24:56.155309 21277 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:35137
I20260807 02:24:56.155336 21410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:35137 every 8 connection(s)
I20260807 02:24:56.156224 21277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:24:56.158298 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 21277
I20260807 02:24:56.158442 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:24:56.158749 4981 external_mini_cluster.cc:1428] Setting key c8a1a7f0a1dde08ad2b978cc051d1e80
Aug 07 02:24:56 dist-test-slave-69rg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069496, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:24:56.170305 20962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:56.158091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:48403 (local address 127.4.221.126:45977)
0807 02:24:56.158282 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:56.158290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:56.159230 (+ 940us) server_negotiation.cc:407] Connection header received
0807 02:24:56.160623 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:56.160626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:56.160685 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:56.160785 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:56.162044 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.162595 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.163750 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.163983 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.164098 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:56.166857 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:56.166860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:56.166862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:56.166886 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:56.168261 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:56.168766 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:56.168768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:56.168770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:56.168848 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:56.169155 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:56.169157 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:56.169158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:56.169358 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:56.169436 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:56.170047 (+ 611us) server_negotiation.cc:299] Negotiation successful
0807 02:24:56.170156 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 02:24:56.171135 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:56.158542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:48403)
0807 02:24:56.159056 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:56.159114 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:56.160409 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:56.160973 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:56.160982 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:56.161369 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:24:56.161907 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:56.161923 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.162733 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.162737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:56.163602 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:56.163612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.164136 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.164141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:56.164319 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:56.164957 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:24:56.164974 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:24:56.166695 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:24:56.168372 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:56.168377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:56.168390 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:56.168651 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:56.168924 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:24:56.168926 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:24:56.168928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:24:56.169069 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:24:56.169476 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:24:56.169481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:24:56.169729 (+ 248us) client_negotiation.cc:769] Sending connection context
0807 02:24:56.169977 (+ 248us) client_negotiation.cc:240] Negotiation successful
0807 02:24:56.170252 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":121,"threads_started":1}
I20260807 02:24:56.172314 21411 heartbeater.cc:347] Connected to a master server at 127.4.221.126:45977
I20260807 02:24:56.172559 21411 heartbeater.cc:464] Registering TS with master...
I20260807 02:24:56.173100 21411 heartbeater.cc:511] Master 127.4.221.126:45977 requested a full tablet report, sending...
I20260807 02:24:56.174139 20920 ts_manager.cc:194] Registered new tserver with Master: c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137)
I20260807 02:24:56.174757 20920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:48403
I20260807 02:24:56.184556 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:24:56.194677 20962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:56.187039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.4.221.126:45977)
0807 02:24:56.187195 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:56.187201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:56.187381 (+ 180us) server_negotiation.cc:407] Connection header received
0807 02:24:56.187539 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:56.187542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:56.187594 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:56.187675 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:24:56.188390 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.188921 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.189782 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.190051 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.190162 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:56.191576 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:24:56.191580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:24:56.191582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:24:56.191607 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:24:56.192898 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:56.193324 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:56.193328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:56.193330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:56.193416 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:24:56.193692 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:24:56.193696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:24:56.193698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:24:56.194006 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:24:56.194145 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:56.194397 (+ 252us) server_negotiation.cc:299] Negotiation successful
0807 02:24:56.194514 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:24:56.197530 20920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:24:56.200999 20920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:24:56.215324 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:56.209543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:44017 (local address 127.0.0.1:48142)
0807 02:24:56.210264 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:56.210278 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:56.210370 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:56.211163 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:56.211165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:56.211181 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:56.211366 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:56.211374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.212452 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.212458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:56.213656 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:56.213666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.214751 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.214755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:56.214865 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:56.214907 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 02:24:56.215026 (+ 119us) client_negotiation.cc:240] Negotiation successful
0807 02:24:56.215140 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":7,"thread_start_us":115,"threads_started":1}
I20260807 02:24:56.215332 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:56.210058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48142 (local address 127.4.221.65:44017)
0807 02:24:56.210598 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:56.210604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:56.210622 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:24:56.210685 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:56.210689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:56.210843 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:56.210990 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:56.211500 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.212332 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.213781 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.214597 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.214745 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:56.214906 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:56.215062 (+ 156us) server_negotiation.cc:299] Negotiation successful
0807 02:24:56.215179 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":102,"threads_started":1}
I20260807 02:24:56.215580 21423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:56.209233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:35137 (local address 127.0.0.1:34990)
0807 02:24:56.209582 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:56.209640 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:56.209954 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:56.210993 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:56.210999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:56.211022 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:56.211159 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:56.211167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.212910 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.212912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:56.214052 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:56.214071 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.215184 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.215188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:56.215277 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:56.215300 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 02:24:56.215369 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 02:24:56.215444 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":62,"thread_start_us":105,"threads_started":1}
I20260807 02:24:56.216012 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:56.209989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:39553 (local address 127.0.0.1:55428)
0807 02:24:56.210622 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:56.210636 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:56.210743 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:56.211306 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:56.211308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:56.211325 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:56.211555 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:56.211562 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.212896 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.212901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:56.214547 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:56.214555 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.215551 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.215554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:56.215658 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:56.215690 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 02:24:56.215759 (+ 69us) client_negotiation.cc:240] Negotiation successful
0807 02:24:56.215859 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":135,"thread_start_us":95,"threads_started":1}
I20260807 02:24:56.216231 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:56.210451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55428 (local address 127.4.221.66:39553)
0807 02:24:56.210873 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:56.210882 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:56.210899 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:24:56.210992 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:56.210997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:56.211174 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:56.211313 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:56.211679 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.212742 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.214669 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.215442 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.215566 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:56.215934 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:56.215994 (+ 60us) server_negotiation.cc:299] Negotiation successful
0807 02:24:56.216090 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":121,"threads_started":1}
I20260807 02:24:56.216779 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:56.209415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.4.221.67:35137)
0807 02:24:56.210509 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:56.210523 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:56.210549 (+ 26us) server_negotiation.cc:407] Connection header received
0807 02:24:56.210625 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:56.210630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:56.210833 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:56.210990 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:56.211400 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.212773 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.214220 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.215049 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.216340 (+ 1291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:56.216483 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:56.216551 (+ 68us) server_negotiation.cc:299] Negotiation successful
0807 02:24:56.216642 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":148,"threads_started":1}
I20260807 02:24:56.219054 21343 tablet_service.cc:1467] Processing CreateTablet for tablet 0059e52dfd1045258a92dd419bc46272 (DEFAULT_TABLE table=test-table [id=5bb094f3c95f429eabe811de7640fe06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:24:56.219091 21053 tablet_service.cc:1467] Processing CreateTablet for tablet 0059e52dfd1045258a92dd419bc46272 (DEFAULT_TABLE table=test-table [id=5bb094f3c95f429eabe811de7640fe06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:24:56.219086 21199 tablet_service.cc:1467] Processing CreateTablet for tablet 0059e52dfd1045258a92dd419bc46272 (DEFAULT_TABLE table=test-table [id=5bb094f3c95f429eabe811de7640fe06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:24:56.220096 21053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0059e52dfd1045258a92dd419bc46272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:56.220096 21199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0059e52dfd1045258a92dd419bc46272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:56.220108 21343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0059e52dfd1045258a92dd419bc46272. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:24:56.226807 21429 tablet_bootstrap.cc:492] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37: Bootstrap starting.
I20260807 02:24:56.227550 21431 tablet_bootstrap.cc:492] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912: Bootstrap starting.
I20260807 02:24:56.229547 21429 tablet_bootstrap.cc:654] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:56.230007 21431 tablet_bootstrap.cc:654] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:56.230175 21430 tablet_bootstrap.cc:492] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: Bootstrap starting.
I20260807 02:24:56.230894 21431 log.cc:824] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:56.231094 21429 log.cc:824] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:56.232362 21430 tablet_bootstrap.cc:654] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: Neither blocks nor log segments found. Creating new log.
I20260807 02:24:56.233151 21430 log.cc:824] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: Log is configured to *not* fsync() on all Append() calls
I20260807 02:24:56.233253 21431 tablet_bootstrap.cc:492] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912: No bootstrap required, opened a new log
I20260807 02:24:56.233448 21431 ts_tablet_manager.cc:1397] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260807 02:24:56.235181 21429 tablet_bootstrap.cc:492] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37: No bootstrap required, opened a new log
I20260807 02:24:56.235181 21430 tablet_bootstrap.cc:492] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: No bootstrap required, opened a new log
I20260807 02:24:56.235375 21429 ts_tablet_manager.cc:1397] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260807 02:24:56.235375 21430 ts_tablet_manager.cc:1397] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 02:24:56.236711 21431 raft_consensus.cc:348] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:24:56.237011 21431 raft_consensus.cc:374] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:56.237087 21431 raft_consensus.cc:729] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db28e04bd78c460284a5f1561c6bb912, State: Initialized, Role: FOLLOWER
I20260807 02:24:56.237615 21431 consensus_queue.cc:260] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:24:56.238739 21431 ts_tablet_manager.cc:1428] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260807 02:24:56.239126 21272 heartbeater.cc:503] Master 127.4.221.126:45977 was elected leader, sending a full tablet report...
I20260807 02:24:56.239702 21429 raft_consensus.cc:348] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:24:56.239789 21430 raft_consensus.cc:348] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:24:56.240027 21429 raft_consensus.cc:374] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:56.240056 21430 raft_consensus.cc:374] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:24:56.240099 21429 raft_consensus.cc:729] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c316b081e1494e54a9c8b520526e4b37, State: Initialized, Role: FOLLOWER
I20260807 02:24:56.240108 21430 raft_consensus.cc:729] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5489a1ae504bc5b0466e9652c1cb0a, State: Initialized, Role: FOLLOWER
I20260807 02:24:56.240547 21429 consensus_queue.cc:260] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:24:56.240547 21430 consensus_queue.cc:260] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:24:56.241519 21430 ts_tablet_manager.cc:1428] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260807 02:24:56.241519 21429 ts_tablet_manager.cc:1428] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260807 02:24:56.241883 21125 heartbeater.cc:503] Master 127.4.221.126:45977 was elected leader, sending a full tablet report...
I20260807 02:24:56.242027 21411 heartbeater.cc:503] Master 127.4.221.126:45977 was elected leader, sending a full tablet report...
W20260807 02:24:56.407246 21412 tablet.cc:2369] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:24:56.421518 21435 raft_consensus.cc:482] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:24:56.421778 21435 raft_consensus.cc:504] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:24:56.423157 21435 leader_election.cc:288] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137), 8c5489a1ae504bc5b0466e9652c1cb0a (127.4.221.65:44017)
W20260807 02:24:56.424613 21126 tablet.cc:2369] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:24:56.427381 21424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:56.423503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:33239 (local address 127.4.221.67:35137)
0807 02:24:56.423652 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:56.423658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:56.424004 (+ 346us) server_negotiation.cc:407] Connection header received
0807 02:24:56.424125 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:56.424128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:56.424177 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:56.424245 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:56.424543 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.425137 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.426371 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.426783 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.426923 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:56.426981 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:56.427166 (+ 185us) server_negotiation.cc:299] Negotiation successful
0807 02:24:56.427251 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 02:24:56.427380 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:56.423439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:44017 (local address 127.4.221.66:45923)
0807 02:24:56.423643 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:56.423660 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:56.423778 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:56.424064 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:56.424070 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:56.424089 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:56.424218 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:56.424247 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.425240 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.425244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:56.426233 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:56.426244 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.426950 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.426954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:56.427073 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:56.427109 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 02:24:56.427192 (+ 83us) client_negotiation.cc:240] Negotiation successful
0807 02:24:56.427262 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":55}
I20260807 02:24:56.427384 21438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:56.423440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:35137 (local address 127.4.221.66:33239)
0807 02:24:56.423893 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:56.423904 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 02:24:56.423996 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:24:56.424288 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:24:56.424294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:24:56.424315 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:24:56.424425 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:56.424430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.425286 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.425289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:56.426231 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:24:56.426241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.426914 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.426918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:24:56.427020 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:56.427050 (+ 30us) client_negotiation.cc:769] Sending connection context
0807 02:24:56.427111 (+ 61us) client_negotiation.cc:240] Negotiation successful
0807 02:24:56.427202 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":57,"thread_start_us":131,"threads_started":1}
I20260807 02:24:56.427423 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:56.423598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:45923 (local address 127.4.221.65:44017)
0807 02:24:56.423756 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:56.423761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:56.423787 (+ 26us) server_negotiation.cc:407] Connection header received
0807 02:24:56.423887 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:56.423889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:56.423932 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:56.424036 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:24:56.424381 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.425100 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.426388 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.426798 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.426933 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:56.426985 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:56.427232 (+ 247us) server_negotiation.cc:299] Negotiation successful
0807 02:24:56.427308 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260807 02:24:56.428064 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059e52dfd1045258a92dd419bc46272" candidate_uuid: "db28e04bd78c460284a5f1561c6bb912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" is_pre_election: true
I20260807 02:24:56.428269 21363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059e52dfd1045258a92dd419bc46272" candidate_uuid: "db28e04bd78c460284a5f1561c6bb912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c316b081e1494e54a9c8b520526e4b37" is_pre_election: true
I20260807 02:24:56.428382 21075 raft_consensus.cc:2454] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db28e04bd78c460284a5f1561c6bb912 in term 0.
I20260807 02:24:56.428572 21363 raft_consensus.cc:2454] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db28e04bd78c460284a5f1561c6bb912 in term 0.
I20260807 02:24:56.428942 21147 leader_election.cc:302] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c5489a1ae504bc5b0466e9652c1cb0a, db28e04bd78c460284a5f1561c6bb912; no voters:
I20260807 02:24:56.429260 21435 raft_consensus.cc:2781] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:24:56.429361 21435 raft_consensus.cc:482] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:24:56.429422 21435 raft_consensus.cc:3037] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:56.430534 21435 raft_consensus.cc:504] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:24:56.430915 21435 leader_election.cc:288] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [CANDIDATE]: Term 1 election: Requested vote from peers c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137), 8c5489a1ae504bc5b0466e9652c1cb0a (127.4.221.65:44017)
I20260807 02:24:56.431361 21363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059e52dfd1045258a92dd419bc46272" candidate_uuid: "db28e04bd78c460284a5f1561c6bb912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c316b081e1494e54a9c8b520526e4b37"
I20260807 02:24:56.431361 21075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0059e52dfd1045258a92dd419bc46272" candidate_uuid: "db28e04bd78c460284a5f1561c6bb912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a"
I20260807 02:24:56.431504 21363 raft_consensus.cc:3037] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:56.431504 21075 raft_consensus.cc:3037] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:24:56.432833 21363 raft_consensus.cc:2454] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db28e04bd78c460284a5f1561c6bb912 in term 1.
I20260807 02:24:56.432833 21075 raft_consensus.cc:2454] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db28e04bd78c460284a5f1561c6bb912 in term 1.
I20260807 02:24:56.433166 21148 leader_election.cc:302] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c316b081e1494e54a9c8b520526e4b37, db28e04bd78c460284a5f1561c6bb912; no voters:
I20260807 02:24:56.433471 21435 raft_consensus.cc:2781] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:24:56.433718 21435 raft_consensus.cc:686] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [term 1 LEADER]: Becoming Leader. State: Replica: db28e04bd78c460284a5f1561c6bb912, State: Running, Role: LEADER
I20260807 02:24:56.434079 21435 consensus_queue.cc:237] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
W20260807 02:24:56.435180 21273 tablet.cc:2369] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:24:56.437371 20914 catalog_manager.cc:5495] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 reported cstate change: term changed from 0 to 1, leader changed from <none> to db28e04bd78c460284a5f1561c6bb912 (127.4.221.66). New cstate: current_term: 1 leader_uuid: "db28e04bd78c460284a5f1561c6bb912" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } health_report { overall_health: UNKNOWN } } }
I20260807 02:24:56.449966 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:56.446726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.4.221.66:39553)
0807 02:24:56.446845 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0807 02:24:56.446848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:24:56.446884 (+ 36us) server_negotiation.cc:407] Connection header received
0807 02:24:56.446949 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:24:56.446951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:24:56.447004 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:24:56.447091 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:24:56.447295 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.447802 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.448621 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:24:56.448835 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:24:56.448909 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:24:56.449155 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:24:56.449567 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:24:56.449670 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 02:24:56.449729 (+ 59us) server_negotiation.cc:299] Negotiation successful
0807 02:24:56.449776 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 02:24:56.455587 21075 raft_consensus.cc:1261] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Refusing update from remote peer db28e04bd78c460284a5f1561c6bb912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:24:56.455583 21363 raft_consensus.cc:1261] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37 [term 1 FOLLOWER]: Refusing update from remote peer db28e04bd78c460284a5f1561c6bb912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:24:56.456323 21439 consensus_queue.cc:1035] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:24:56.456511 21435 consensus_queue.cc:1035] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:24:56.467512 21445 mvcc.cc:205] Tried to move back new op lower bound from 7315740657474985984 to 7315740657396789248. Current Snapshot: MvccSnapshot[applied={T|T < 7315740657474985984}]
I20260807 02:24:56.473850 20918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:24:56.474058 20918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:24:56.476859 20918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Sending DeleteTablet for 3 replicas of tablet 0059e52dfd1045258a92dd419bc46272
I20260807 02:24:56.477895 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 20987
I20260807 02:24:56.477919 21343 tablet_service.cc:1514] Processing DeleteTablet for tablet 0059e52dfd1045258a92dd419bc46272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:24:56 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:34990
I20260807 02:24:56.478814 21199 tablet_service.cc:1514] Processing DeleteTablet for tablet 0059e52dfd1045258a92dd419bc46272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:24:56 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:55428
I20260807 02:24:56.479645 21454 tablet_replica.cc:330] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912: stopping tablet replica
I20260807 02:24:56.480096 21454 raft_consensus.cc:2229] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:24:56.480612 21454 raft_consensus.cc:2258] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:24:56.481212 21453 tablet_replica.cc:330] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37: stopping tablet replica
I20260807 02:24:56.481570 21453 raft_consensus.cc:2229] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:24:56.481966 21453 raft_consensus.cc:2258] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:24:56.482506 21454 ts_tablet_manager.cc:1905] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:24:56.483495 21453 ts_tablet_manager.cc:1905] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:24:56.485777 21454 ts_tablet_manager.cc:1918] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:24:56.485956 21454 log.cc:1196] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/0059e52dfd1045258a92dd419bc46272
I20260807 02:24:56.486389 21454 ts_tablet_manager.cc:1939] T 0059e52dfd1045258a92dd419bc46272 P db28e04bd78c460284a5f1561c6bb912: Deleting consensus metadata
I20260807 02:24:56.486372 21453 ts_tablet_manager.cc:1918] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:24:56.486465 21453 log.cc:1196] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/0059e52dfd1045258a92dd419bc46272
I20260807 02:24:56.486814 21453 ts_tablet_manager.cc:1939] T 0059e52dfd1045258a92dd419bc46272 P c316b081e1494e54a9c8b520526e4b37: Deleting consensus metadata
I20260807 02:24:56.487722 20905 catalog_manager.cc:4841] TS db28e04bd78c460284a5f1561c6bb912 (127.4.221.66:39553): tablet 0059e52dfd1045258a92dd419bc46272 (table test-table [id=5bb094f3c95f429eabe811de7640fe06]) successfully deleted
I20260807 02:24:56.487983 20905 catalog_manager.cc:4841] TS c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137): tablet 0059e52dfd1045258a92dd419bc46272 (table test-table [id=5bb094f3c95f429eabe811de7640fe06]) successfully deleted
W20260807 02:24:56.488026 20904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:44017 (error 108)
I20260807 02:24:56.488335 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 21130
I20260807 02:24:56.490857 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:24:56.490535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:44017 (local address 127.0.0.1:48156)
0807 02:24:56.490672 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0807 02:24:56.490766 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:44017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 02:24:56.491014 20904 catalog_manager.cc:4584] TS 8c5489a1ae504bc5b0466e9652c1cb0a (127.4.221.65:44017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0059e52dfd1045258a92dd419bc46272: Network error: Client connection negotiation failed: client connection to 127.4.221.65:44017: connect: Connection refused (error 111)
I20260807 02:24:56.495766 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 21277
I20260807 02:24:56.503252 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 20887
I20260807 02:25:16.511498 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:45977
--webserver_interface=127.4.221.126
--webserver_port=43253
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:45977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:25:16.649248 21456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:16.649685 21456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:16.649770 21456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:16.654292 21456 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 02:25:16.654422 21456 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 02:25:16.654450 21456 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 02:25:16.654476 21456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:25:16.654531 21456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:16.654594 21456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:16.654659 21456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:25:16.654706 21456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:25:16.661141 21456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:45977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:45977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:16.662753 21456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:16.664108 21456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:16.682700 21463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:16.682689 21462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:16.682763 21456 server_base.cc:1034] running on GCE node
W20260807 02:25:16.682689 21465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:16.683884 21456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:16.685612 21456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:16.686882 21456 hybrid_clock.cc:648] HybridClock initialized: now 1786069516686843 us; error 44 us; skew 500 ppm
Aug 07 02:25:16 dist-test-slave-69rg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069516, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:16.690706 21456 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:25:16.692065 21456 webserver.cc:460] Webserver started at http://127.4.221.126:43253/ using document root <none> and password file <none>
I20260807 02:25:16.692866 21456 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:16.693015 21456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:16.698210 21456 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:25:16.701042 21472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:16.702813 21456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:25:16.703001 21456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "f56a6067d16f482a90c0d0c5f75382f6"
format_stamp: "Formatted at 2026-08-07 02:24:55 on dist-test-slave-69rg"
server_key: "93e2e995ff605092a9af472331b22272"
server_key_iv: "484c7ef14169b8983b753471dffbc885"
server_key_version: "encryptionkey@0"
I20260807 02:25:16.703836 21456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:16.731194 21456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:16.735487 21456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:16.735762 21456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:16.745065 21456 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:45977
I20260807 02:25:16.745419 21528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:45977 every 8 connection(s)
I20260807 02:25:16.746634 21456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:25:16.752386 21529 sys_catalog.cc:263] Verifying existing consensus state
I20260807 02:25:16.752667 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 21456
I20260807 02:25:16.753262 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:44017
--local_ip_for_outbound_sockets=127.4.221.65
--tserver_master_addrs=127.4.221.126:45977
--webserver_port=34421
--webserver_interface=127.4.221.65
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:25:16.755321 21529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Bootstrap starting.
I20260807 02:25:16.777452 21529 log.cc:824] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:16.787974 21529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:25:16.788493 21529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Bootstrap complete.
I20260807 02:25:16.793457 21529 raft_consensus.cc:348] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } }
I20260807 02:25:16.794046 21529 raft_consensus.cc:729] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56a6067d16f482a90c0d0c5f75382f6, State: Initialized, Role: FOLLOWER
I20260807 02:25:16.794740 21529 consensus_queue.cc:260] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } }
I20260807 02:25:16.794876 21529 raft_consensus.cc:388] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:25:16.794981 21529 raft_consensus.cc:482] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:25:16.795094 21529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 1 FOLLOWER]: Advancing to term 2
I20260807 02:25:16.797644 21529 raft_consensus.cc:504] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } }
I20260807 02:25:16.798128 21529 leader_election.cc:302] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f56a6067d16f482a90c0d0c5f75382f6; no voters:
I20260807 02:25:16.798597 21529 leader_election.cc:288] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260807 02:25:16.798710 21534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 2 FOLLOWER]: Leader election won for term 2
I20260807 02:25:16.798950 21534 raft_consensus.cc:686] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 2 LEADER]: Becoming Leader. State: Replica: f56a6067d16f482a90c0d0c5f75382f6, State: Running, Role: LEADER
I20260807 02:25:16.799427 21534 consensus_queue.cc:237] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } }
I20260807 02:25:16.800220 21529 sys_catalog.cc:564] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:25:16.801375 21536 sys_catalog.cc:455] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f56a6067d16f482a90c0d0c5f75382f6. Latest consensus state: current_term: 2 leader_uuid: "f56a6067d16f482a90c0d0c5f75382f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } } }
I20260807 02:25:16.801437 21535 sys_catalog.cc:455] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f56a6067d16f482a90c0d0c5f75382f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } } }
I20260807 02:25:16.801532 21536 sys_catalog.cc:458] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [sys.catalog]: This master's current role is: LEADER
I20260807 02:25:16.801566 21535 sys_catalog.cc:458] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [sys.catalog]: This master's current role is: LEADER
I20260807 02:25:16.801993 21543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:25:16.808208 21543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:25:16.809222 21543 catalog_manager.cc:1259] Loaded cluster ID: d385d383e608403193d5e65db8fee151
I20260807 02:25:16.809298 21543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:25:16.812178 21543 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:25:16.813041 21543 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Loaded TSK: 0
I20260807 02:25:16.830309 21543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Generated new TSK 1
I20260807 02:25:16.830693 21543 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 02:25:16.898440 21532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:16.898841 21532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:16.898928 21532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:16.904663 21532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:16.904816 21532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:16.904937 21532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:25:16.912403 21532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:44017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=34421
--tserver_master_addrs=127.4.221.126:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:16.914239 21532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:16.915731 21532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:16.930462 21561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:16.930558 21558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:16.930462 21559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:16.931137 21532 server_base.cc:1034] running on GCE node
I20260807 02:25:16.931700 21532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:16.932735 21532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:16.933926 21532 hybrid_clock.cc:648] HybridClock initialized: now 1786069516933901 us; error 74 us; skew 500 ppm
Aug 07 02:25:16 dist-test-slave-69rg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069516, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:16.942444 21532 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:25:16.943749 21532 webserver.cc:460] Webserver started at http://127.4.221.65:34421/ using document root <none> and password file <none>
I20260807 02:25:16.944484 21532 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:16.944626 21532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:16.949807 21532 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:25:16.952716 21569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:16.954425 21532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:25:16.954607 21532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "8c5489a1ae504bc5b0466e9652c1cb0a"
format_stamp: "Formatted at 2026-08-07 02:24:55 on dist-test-slave-69rg"
server_key: "cca6f77050a51fd3a77688a33d438748"
server_key_iv: "8536e69d0f6a730b4c95ad29b574b459"
server_key_version: "encryptionkey@0"
I20260807 02:25:16.955305 21532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:16.971935 21532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:16.975901 21532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:16.976145 21532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:16.976897 21532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:25:16.978261 21578 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 02:25:16.983590 21532 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 02:25:16.983711 21532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260807 02:25:16.983819 21532 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 02:25:16.986222 21532 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 02:25:16.986299 21532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260807 02:25:16.986675 21578 tablet_bootstrap.cc:492] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: Bootstrap starting.
I20260807 02:25:16.998908 21532 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:44017
I20260807 02:25:16.998983 21689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:44017 every 8 connection(s)
I20260807 02:25:17.000137 21532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:25:17.004640 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 21532
I20260807 02:25:17.005266 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:39553
--local_ip_for_outbound_sockets=127.4.221.66
--tserver_master_addrs=127.4.221.126:45977
--webserver_port=37791
--webserver_interface=127.4.221.66
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:25:17.012108 21578 log.cc:824] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: Log is configured to *not* fsync() on all Append() calls
Aug 07 02:25:17 dist-test-slave-69rg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069516, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:25:17.022689 21578 tablet_bootstrap.cc:492] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:25:17.023321 21578 tablet_bootstrap.cc:492] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: Bootstrap complete.
I20260807 02:25:17.024256 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.003184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:46477)
0807 02:25:17.003891 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:17.003966 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:17.005089 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:17.006630 (+ 1541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:17.006639 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:17.007187 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:25:17.007838 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.007884 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.010188 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.010192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.011162 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.011174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.012460 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.012465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.012697 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.013589 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:25:17.013615 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:25:17.016074 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:25:17.019707 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:17.019714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:17.019729 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:17.020139 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:17.020706 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:17.020710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:17.020712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:17.020906 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:17.021903 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:25:17.021910 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:25:17.022234 (+ 324us) client_negotiation.cc:769] Sending connection context
0807 02:25:17.022461 (+ 227us) client_negotiation.cc:240] Negotiation successful
0807 02:25:17.023020 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":162,"threads_started":1}
I20260807 02:25:17.024250 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.003184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:46477 (local address 127.4.221.126:45977)
0807 02:25:17.003772 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:17.003804 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:17.004126 (+ 322us) server_negotiation.cc:407] Connection header received
0807 02:25:17.005612 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:17.005625 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:17.006136 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:17.006572 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:17.008155 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.009985 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.011321 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.012189 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.012409 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.016410 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:17.016421 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:17.016447 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:17.016503 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:17.019509 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:17.020307 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:17.020317 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:17.020328 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:17.020560 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:17.021027 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:17.021034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:17.021038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:17.021700 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:17.021932 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:17.022652 (+ 720us) server_negotiation.cc:299] Negotiation successful
0807 02:25:17.023109 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":179,"threads_started":1}
I20260807 02:25:17.025269 21578 ts_tablet_manager.cc:1397] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: Time spent bootstrapping tablet: real 0.039s user 0.032s sys 0.005s
I20260807 02:25:17.028435 21690 heartbeater.cc:347] Connected to a master server at 127.4.221.126:45977
I20260807 02:25:17.028752 21690 heartbeater.cc:464] Registering TS with master...
I20260807 02:25:17.028813 21578 raft_consensus.cc:348] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:17.029472 21578 raft_consensus.cc:729] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5489a1ae504bc5b0466e9652c1cb0a, State: Initialized, Role: FOLLOWER
I20260807 02:25:17.029513 21690 heartbeater.cc:511] Master 127.4.221.126:45977 requested a full tablet report, sending...
I20260807 02:25:17.030225 21578 consensus_queue.cc:260] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:17.031580 21578 ts_tablet_manager.cc:1428] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260807 02:25:17.034085 21491 ts_manager.cc:194] Registered new tserver with Master: 8c5489a1ae504bc5b0466e9652c1cb0a (127.4.221.65:44017)
I20260807 02:25:17.039718 21491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:46477
I20260807 02:25:17.042999 21690 heartbeater.cc:503] Master 127.4.221.126:45977 was elected leader, sending a full tablet report...
Aug 07 02:25:17 dist-test-slave-69rg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069516, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for kudu/127.4.221.65@KRBTEST.COM
I20260807 02:25:17.052415 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.039184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56320 (local address 127.4.221.65:44017)
0807 02:25:17.039748 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:17.039763 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:17.040261 (+ 498us) server_negotiation.cc:407] Connection header received
0807 02:25:17.040544 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:17.040549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:17.040752 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:17.040937 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:17.041925 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.043110 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.044511 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.045061 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.045188 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.048269 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:17.048280 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:17.048292 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:17.048320 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:17.050229 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:17.050783 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:17.050789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:17.050790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:17.050874 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:17.051185 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:17.051190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:17.051192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:17.051495 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:17.051648 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:17.052034 (+ 386us) server_negotiation.cc:299] Negotiation successful
0807 02:25:17.052233 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":131,"threads_started":1}
I20260807 02:25:17.052415 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.039037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:44017 (local address 127.0.0.1:56320)
0807 02:25:17.040084 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:17.040130 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:17.040344 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:17.040958 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:17.040965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:17.041514 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:25:17.041692 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.041704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.043286 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.043290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.044337 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.044352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.045191 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.045195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.045324 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.046129 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:25:17.046142 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:25:17.048091 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:25:17.050358 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:17.050369 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:17.050376 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:17.050649 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:17.050991 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:17.050995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:17.050997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:17.051090 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:17.051616 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:25:17.051623 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:25:17.051771 (+ 148us) client_negotiation.cc:769] Sending connection context
0807 02:25:17.051927 (+ 156us) client_negotiation.cc:240] Negotiation successful
0807 02:25:17.052185 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"thread_start_us":190,"threads_started":1}
I20260807 02:25:17.053704 21621 tablet_service.cc:1514] Processing DeleteTablet for tablet 0059e52dfd1045258a92dd419bc46272 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:24:56 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:56320
I20260807 02:25:17.055186 21699 tablet_replica.cc:330] stopping tablet replica
I20260807 02:25:17.055573 21699 raft_consensus.cc:2229] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:25:17.055894 21699 raft_consensus.cc:2258] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:25:17.057168 21699 ts_tablet_manager.cc:1905] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:25:17.061949 21699 ts_tablet_manager.cc:1918] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:25:17.062098 21699 log.cc:1196] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal/wals/0059e52dfd1045258a92dd419bc46272
I20260807 02:25:17.062413 21699 ts_tablet_manager.cc:1939] T 0059e52dfd1045258a92dd419bc46272 P 8c5489a1ae504bc5b0466e9652c1cb0a: Deleting consensus metadata
I20260807 02:25:17.064786 21473 catalog_manager.cc:4841] TS 8c5489a1ae504bc5b0466e9652c1cb0a (127.4.221.65:44017): tablet 0059e52dfd1045258a92dd419bc46272 (table test-table [id=5bb094f3c95f429eabe811de7640fe06]) successfully deleted
W20260807 02:25:17.154754 21694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:17.155050 21694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:17.155095 21694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:17.159418 21694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:17.159497 21694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:17.159588 21694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:25:17.164995 21694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:39553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=37791
--tserver_master_addrs=127.4.221.126:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:17.166234 21694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:17.167559 21694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:17.177907 21704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:17.177908 21705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:17.177912 21707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:17.178350 21694 server_base.cc:1034] running on GCE node
I20260807 02:25:17.178814 21694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:17.179529 21694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:17.180742 21694 hybrid_clock.cc:648] HybridClock initialized: now 1786069517180724 us; error 36 us; skew 500 ppm
Aug 07 02:25:17 dist-test-slave-69rg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069517, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:17.183979 21694 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:25:17.184983 21694 webserver.cc:460] Webserver started at http://127.4.221.66:37791/ using document root <none> and password file <none>
I20260807 02:25:17.185624 21694 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:17.185708 21694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:17.190354 21694 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:25:17.192677 21714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:17.194273 21694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:25:17.194406 21694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "db28e04bd78c460284a5f1561c6bb912"
format_stamp: "Formatted at 2026-08-07 02:24:55 on dist-test-slave-69rg"
server_key: "f0da3b3baebbeaf8963a1948416145bd"
server_key_iv: "c44208ff5c2649179752e03534af1a4b"
server_key_version: "encryptionkey@0"
I20260807 02:25:17.195001 21694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:17.214833 21694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:17.218575 21694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:17.218804 21694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:17.219499 21694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:25:17.220407 21694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:25:17.220458 21694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:17.220528 21694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:25:17.220570 21694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:17.230942 21694 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:39553
I20260807 02:25:17.230974 21831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:39553 every 8 connection(s)
I20260807 02:25:17.232043 21694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:25:17.238781 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 21694
I20260807 02:25:17.239364 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:35137
--local_ip_for_outbound_sockets=127.4.221.67
--tserver_master_addrs=127.4.221.126:45977
--webserver_port=40017
--webserver_interface=127.4.221.67
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 07 02:25:17 dist-test-slave-69rg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069517, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:25:17.248639 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.234224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:57361 (local address 127.4.221.126:45977)
0807 02:25:17.234392 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:17.234396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:17.235324 (+ 928us) server_negotiation.cc:407] Connection header received
0807 02:25:17.236612 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:17.236615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:17.236669 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:17.236750 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:17.238061 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.238767 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.240583 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.240851 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.240947 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.244074 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:17.244079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:17.244082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:17.244117 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:17.246030 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:17.246737 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:17.246740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:17.246742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:17.246826 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:17.247289 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:17.247291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:17.247292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:17.247478 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:17.247555 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:17.248349 (+ 794us) server_negotiation.cc:299] Negotiation successful
0807 02:25:17.248490 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 02:25:17.250247 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.234635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:57361)
0807 02:25:17.235138 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:17.235194 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:17.236380 (+ 1186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:17.236927 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:17.236934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:17.237315 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:25:17.237887 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.237908 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.238919 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.238924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.240433 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.240448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.241035 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.241042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.241285 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.242097 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:25:17.242114 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:25:17.243791 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:25:17.246173 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:17.246182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:17.246205 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:17.246626 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:17.246927 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:17.246932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:17.246934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:17.247152 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:17.247592 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:25:17.247601 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:25:17.247995 (+ 394us) client_negotiation.cc:769] Sending connection context
0807 02:25:17.248270 (+ 275us) client_negotiation.cc:240] Negotiation successful
0807 02:25:17.248705 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":123,"threads_started":1}
I20260807 02:25:17.251643 21832 heartbeater.cc:347] Connected to a master server at 127.4.221.126:45977
I20260807 02:25:17.251946 21832 heartbeater.cc:464] Registering TS with master...
I20260807 02:25:17.252668 21832 heartbeater.cc:511] Master 127.4.221.126:45977 requested a full tablet report, sending...
I20260807 02:25:17.254060 21491 ts_manager.cc:194] Registered new tserver with Master: db28e04bd78c460284a5f1561c6bb912 (127.4.221.66:39553)
I20260807 02:25:17.254717 21491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:57361
W20260807 02:25:17.368727 21835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:17.369050 21835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:17.369131 21835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:17.373291 21835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:17.373391 21835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:17.373489 21835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:25:17.378968 21835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:35137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=40017
--tserver_master_addrs=127.4.221.126:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:17.380088 21835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:17.381160 21835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:17.391031 21843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:17.391093 21840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:17.391034 21841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:17.391268 21835 server_base.cc:1034] running on GCE node
I20260807 02:25:17.391917 21835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:17.392550 21835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:17.393776 21835 hybrid_clock.cc:648] HybridClock initialized: now 1786069517393757 us; error 34 us; skew 500 ppm
Aug 07 02:25:17 dist-test-slave-69rg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069517, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:17.396601 21835 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:25:17.397537 21835 webserver.cc:460] Webserver started at http://127.4.221.67:40017/ using document root <none> and password file <none>
I20260807 02:25:17.398151 21835 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:17.398233 21835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:17.402513 21835 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:25:17.404572 21851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:17.406322 21835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:25:17.406451 21835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "c316b081e1494e54a9c8b520526e4b37"
format_stamp: "Formatted at 2026-08-07 02:24:56 on dist-test-slave-69rg"
server_key: "e28b8dda8bf7caa0f89352e62f3734aa"
server_key_iv: "0c8487685afa346a61745d6affa13ff4"
server_key_version: "encryptionkey@0"
I20260807 02:25:17.407011 21835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:17.424428 21835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:17.427536 21835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:17.427765 21835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:17.428400 21835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:25:17.429319 21835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:25:17.429369 21835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:17.429441 21835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:25:17.429478 21835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:17.438978 21835 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:35137
I20260807 02:25:17.438997 21970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:35137 every 8 connection(s)
I20260807 02:25:17.440052 21835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:25:17.449573 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 21835
Aug 07 02:25:17 dist-test-slave-69rg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069517, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:25:17.454491 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.441911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:55167 (local address 127.4.221.126:45977)
0807 02:25:17.442071 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:17.442076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:17.443026 (+ 950us) server_negotiation.cc:407] Connection header received
0807 02:25:17.444317 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:17.444319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:17.444371 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:17.444448 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:17.445617 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.446401 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.447523 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.447764 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.447850 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.450490 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:17.450494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:17.450496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:17.450528 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:17.452277 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:17.452824 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:17.452827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:17.452828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:17.452907 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:17.453298 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:17.453300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:17.453301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:17.453467 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:17.453543 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:17.454199 (+ 656us) server_negotiation.cc:299] Negotiation successful
0807 02:25:17.454321 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 02:25:17.455382 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.442359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:55167)
0807 02:25:17.442841 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:17.442896 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:17.444104 (+ 1208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:17.444623 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:17.444630 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:17.445009 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:25:17.445485 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.445498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.446521 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.446524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.447409 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.447419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.447902 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.447905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.448044 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.448655 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:25:17.448671 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:25:17.450320 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:25:17.452399 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:17.452405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:17.452422 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:17.452722 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:17.452993 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:17.452996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:17.452998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:17.453199 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:17.453572 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:25:17.453579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:25:17.453948 (+ 369us) client_negotiation.cc:769] Sending connection context
0807 02:25:17.454146 (+ 198us) client_negotiation.cc:240] Negotiation successful
0807 02:25:17.454515 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":124,"threads_started":1}
I20260807 02:25:17.456884 21971 heartbeater.cc:347] Connected to a master server at 127.4.221.126:45977
I20260807 02:25:17.457144 21971 heartbeater.cc:464] Registering TS with master...
I20260807 02:25:17.457664 21971 heartbeater.cc:511] Master 127.4.221.126:45977 requested a full tablet report, sending...
I20260807 02:25:17.458837 21491 ts_manager.cc:194] Registered new tserver with Master: c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137)
I20260807 02:25:17.459074 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.450369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37956 (local address 127.4.221.126:45977)
0807 02:25:17.450651 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:17.450655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:17.450819 (+ 164us) server_negotiation.cc:407] Connection header received
0807 02:25:17.450937 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:17.450941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:17.450998 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:17.451094 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:17.452094 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.452637 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.453755 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.454051 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.454163 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.455668 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:17.455672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:17.455674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:17.455701 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:17.457393 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:17.457806 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:17.457810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:17.457833 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:17.457915 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:17.458213 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:17.458218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:17.458220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:17.458398 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:17.458629 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:17.458755 (+ 126us) server_negotiation.cc:299] Negotiation successful
0807 02:25:17.458895 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":97,"threads_started":1}
I20260807 02:25:17.459553 21491 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:55167
I20260807 02:25:17.460978 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:25:17.464890 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.461623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37970 (local address 127.4.221.126:45977)
0807 02:25:17.461724 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:17.461729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:17.461943 (+ 214us) server_negotiation.cc:407] Connection header received
0807 02:25:17.462073 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:17.462075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:17.462159 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:17.462236 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:25:17.462531 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.463011 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.463885 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.464098 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.464173 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.464464 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:25:17.464595 (+ 131us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0807 02:25:17.464793 (+ 198us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37970: token expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260807 02:25:17.464970 21975 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37970: token expired
W20260807 02:25:17.465389 21416 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.221.126:45977)
I20260807 02:25:17.472608 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.465793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37972 (local address 127.4.221.126:45977)
0807 02:25:17.465933 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:17.465937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:17.465973 (+ 36us) server_negotiation.cc:407] Connection header received
0807 02:25:17.466060 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:17.466063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:17.466096 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:17.466156 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:17.466763 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.467236 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.468034 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.468225 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.468320 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.469653 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:17.469656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:17.469659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:17.469681 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:17.471211 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:17.471632 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:17.471634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:17.471635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:17.471708 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:17.471944 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:17.471946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:17.471947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:17.472100 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:17.472197 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:17.472397 (+ 200us) server_negotiation.cc:299] Negotiation successful
0807 02:25:17.472491 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260807 02:25:17.475152 21491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:25:17.478713 21491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:25:17.487385 21621 tablet_service.cc:1467] Processing CreateTablet for tablet 4424f2bc44bc46c89a23a81846afc90a (DEFAULT_TABLE table=test-table [id=e52e6b002c6d47ac897b1c29ab06449e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:25:17.488173 21621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4424f2bc44bc46c89a23a81846afc90a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:25:17.491667 21578 tablet_bootstrap.cc:492] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a: Bootstrap starting.
I20260807 02:25:17.491873 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.486191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:39553 (local address 127.0.0.1:39340)
0807 02:25:17.486317 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:17.486337 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:17.486418 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:17.487610 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:17.487613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:17.487632 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:17.487755 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.487762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.489294 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.489298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.490263 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.490274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.491371 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.491375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.491482 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.491544 (+ 62us) client_negotiation.cc:769] Sending connection context
0807 02:25:17.491605 (+ 61us) client_negotiation.cc:240] Negotiation successful
0807 02:25:17.491724 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260807 02:25:17.492614 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.486392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39340 (local address 127.4.221.66:39553)
0807 02:25:17.487061 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:17.487074 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:17.487098 (+ 24us) server_negotiation.cc:407] Connection header received
0807 02:25:17.487180 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:17.487186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:17.487418 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:17.487919 (+ 501us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:17.488110 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.489169 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.490394 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.491233 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.492149 (+ 916us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.492292 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:17.492347 (+ 55us) server_negotiation.cc:299] Negotiation successful
0807 02:25:17.492463 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":115,"threads_started":1}
I20260807 02:25:17.492607 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.486450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:35137 (local address 127.0.0.1:37752)
0807 02:25:17.486792 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:17.486806 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:17.486904 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:17.487791 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:17.487793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:17.487808 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:17.487880 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.487884 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.489474 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.489479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.490583 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.490595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.492220 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.492223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.492319 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.492342 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 02:25:17.492419 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 02:25:17.492487 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":101,"threads_started":1}
I20260807 02:25:17.492643 21578 tablet_bootstrap.cc:654] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a: Neither blocks nor log segments found. Creating new log.
I20260807 02:25:17.492772 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.486744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.4.221.67:35137)
0807 02:25:17.487179 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:17.487188 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:17.487206 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:25:17.487266 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:17.487270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:17.487467 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:17.487641 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:17.488020 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.489346 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.490726 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.491859 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.491967 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.492139 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:17.492483 (+ 344us) server_negotiation.cc:299] Negotiation successful
0807 02:25:17.492610 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":141,"threads_started":1}
I20260807 02:25:17.494704 21578 tablet_bootstrap.cc:492] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a: No bootstrap required, opened a new log
I20260807 02:25:17.494809 21578 ts_tablet_manager.cc:1397] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260807 02:25:17.495301 21578 raft_consensus.cc:348] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:17.495427 21578 raft_consensus.cc:374] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:25:17.495465 21578 raft_consensus.cc:729] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5489a1ae504bc5b0466e9652c1cb0a, State: Initialized, Role: FOLLOWER
I20260807 02:25:17.495604 21578 consensus_queue.cc:260] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:17.495862 21578 ts_tablet_manager.cc:1428] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260807 02:25:17.495724 21901 tablet_service.cc:1467] Processing CreateTablet for tablet 4424f2bc44bc46c89a23a81846afc90a (DEFAULT_TABLE table=test-table [id=e52e6b002c6d47ac897b1c29ab06449e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:25:17.495733 21763 tablet_service.cc:1467] Processing CreateTablet for tablet 4424f2bc44bc46c89a23a81846afc90a (DEFAULT_TABLE table=test-table [id=e52e6b002c6d47ac897b1c29ab06449e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:25:17.496862 21763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4424f2bc44bc46c89a23a81846afc90a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:25:17.496865 21901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4424f2bc44bc46c89a23a81846afc90a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260807 02:25:17.501585 21691 tablet.cc:2369] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:25:17.504812 21985 tablet_bootstrap.cc:492] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37: Bootstrap starting.
I20260807 02:25:17.505426 21984 tablet_bootstrap.cc:492] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912: Bootstrap starting.
I20260807 02:25:17.508097 21985 tablet_bootstrap.cc:654] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37: Neither blocks nor log segments found. Creating new log.
I20260807 02:25:17.508556 21984 tablet_bootstrap.cc:654] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912: Neither blocks nor log segments found. Creating new log.
I20260807 02:25:17.509497 21984 log.cc:824] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:17.509497 21985 log.cc:824] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:17.512051 21984 tablet_bootstrap.cc:492] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912: No bootstrap required, opened a new log
I20260807 02:25:17.512051 21985 tablet_bootstrap.cc:492] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37: No bootstrap required, opened a new log
I20260807 02:25:17.512279 21984 ts_tablet_manager.cc:1397] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260807 02:25:17.512279 21985 ts_tablet_manager.cc:1397] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260807 02:25:17.516498 21985 raft_consensus.cc:348] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:17.516498 21984 raft_consensus.cc:348] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:17.516837 21984 raft_consensus.cc:374] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:25:17.516837 21985 raft_consensus.cc:374] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:25:17.516894 21984 raft_consensus.cc:729] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db28e04bd78c460284a5f1561c6bb912, State: Initialized, Role: FOLLOWER
I20260807 02:25:17.516902 21985 raft_consensus.cc:729] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c316b081e1494e54a9c8b520526e4b37, State: Initialized, Role: FOLLOWER
I20260807 02:25:17.517437 21984 consensus_queue.cc:260] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:17.517436 21985 consensus_queue.cc:260] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:17.518291 21971 heartbeater.cc:503] Master 127.4.221.126:45977 was elected leader, sending a full tablet report...
I20260807 02:25:17.518659 21984 ts_tablet_manager.cc:1428] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260807 02:25:17.518659 21985 ts_tablet_manager.cc:1428] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260807 02:25:17.519126 21832 heartbeater.cc:503] Master 127.4.221.126:45977 was elected leader, sending a full tablet report...
I20260807 02:25:17.529762 21989 raft_consensus.cc:482] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:25:17.530010 21989 raft_consensus.cc:504] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:17.531199 21989 leader_election.cc:288] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137), 8c5489a1ae504bc5b0466e9652c1cb0a (127.4.221.65:44017)
I20260807 02:25:17.535125 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.531530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:36119 (local address 127.4.221.67:35137)
0807 02:25:17.531679 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:17.531684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:17.531740 (+ 56us) server_negotiation.cc:407] Connection header received
0807 02:25:17.531791 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:17.531793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:17.531834 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:17.531891 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:17.532229 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.532993 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.534118 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.534529 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.534665 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.534732 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:17.534923 (+ 191us) server_negotiation.cc:299] Negotiation successful
0807 02:25:17.534993 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 02:25:17.535125 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.531433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:35137 (local address 127.4.221.66:36119)
0807 02:25:17.531541 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:17.531558 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:17.531662 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:17.531938 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:17.531941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:17.531962 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:17.532091 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.532097 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.533120 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.533123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.533998 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.534007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.534660 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.534664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.534782 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.534805 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 02:25:17.534905 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 02:25:17.534984 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260807 02:25:17.535501 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.531435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:44017 (local address 127.4.221.66:37027)
0807 02:25:17.532169 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:17.532184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:17.532326 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:17.532588 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:17.532591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:17.532609 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:17.532723 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.532730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.533488 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.533494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.534366 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:17.534375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.535108 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.535111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:17.535213 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.535235 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 02:25:17.535297 (+ 62us) client_negotiation.cc:240] Negotiation successful
0807 02:25:17.535366 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":287,"thread_start_us":173,"threads_started":1}
I20260807 02:25:17.535526 21698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.531580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:37027 (local address 127.4.221.65:44017)
0807 02:25:17.531731 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:17.531736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:17.532309 (+ 573us) server_negotiation.cc:407] Connection header received
0807 02:25:17.532408 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:17.532410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:17.532469 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:17.532546 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:17.532834 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.533370 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.534504 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.534980 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.535118 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.535295 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:17.535335 (+ 40us) server_negotiation.cc:299] Negotiation successful
0807 02:25:17.535410 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 02:25:17.535921 21923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4424f2bc44bc46c89a23a81846afc90a" candidate_uuid: "db28e04bd78c460284a5f1561c6bb912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c316b081e1494e54a9c8b520526e4b37" is_pre_election: true
I20260807 02:25:17.536198 21642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4424f2bc44bc46c89a23a81846afc90a" candidate_uuid: "db28e04bd78c460284a5f1561c6bb912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" is_pre_election: true
I20260807 02:25:17.536262 21923 raft_consensus.cc:2454] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db28e04bd78c460284a5f1561c6bb912 in term 0.
I20260807 02:25:17.536490 21642 raft_consensus.cc:2454] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db28e04bd78c460284a5f1561c6bb912 in term 0.
I20260807 02:25:17.536795 21716 leader_election.cc:302] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c316b081e1494e54a9c8b520526e4b37, db28e04bd78c460284a5f1561c6bb912; no voters:
I20260807 02:25:17.537118 21989 raft_consensus.cc:2781] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:25:17.537204 21989 raft_consensus.cc:482] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:25:17.537271 21989 raft_consensus.cc:3037] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:25:17.538560 21989 raft_consensus.cc:504] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:17.538902 21989 leader_election.cc:288] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [CANDIDATE]: Term 1 election: Requested vote from peers c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137), 8c5489a1ae504bc5b0466e9652c1cb0a (127.4.221.65:44017)
I20260807 02:25:17.539259 21923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4424f2bc44bc46c89a23a81846afc90a" candidate_uuid: "db28e04bd78c460284a5f1561c6bb912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c316b081e1494e54a9c8b520526e4b37"
I20260807 02:25:17.539273 21642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4424f2bc44bc46c89a23a81846afc90a" candidate_uuid: "db28e04bd78c460284a5f1561c6bb912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a"
I20260807 02:25:17.539379 21923 raft_consensus.cc:3037] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:25:17.539412 21642 raft_consensus.cc:3037] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:25:17.540480 21923 raft_consensus.cc:2454] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db28e04bd78c460284a5f1561c6bb912 in term 1.
I20260807 02:25:17.540484 21642 raft_consensus.cc:2454] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db28e04bd78c460284a5f1561c6bb912 in term 1.
I20260807 02:25:17.540793 21716 leader_election.cc:302] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c316b081e1494e54a9c8b520526e4b37, db28e04bd78c460284a5f1561c6bb912; no voters:
I20260807 02:25:17.541030 21989 raft_consensus.cc:2781] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:25:17.541296 21989 raft_consensus.cc:686] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [term 1 LEADER]: Becoming Leader. State: Replica: db28e04bd78c460284a5f1561c6bb912, State: Running, Role: LEADER
I20260807 02:25:17.541676 21989 consensus_queue.cc:237] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:17.545070 21491 catalog_manager.cc:5495] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 reported cstate change: term changed from 0 to 1, leader changed from <none> to db28e04bd78c460284a5f1561c6bb912 (127.4.221.66). New cstate: current_term: 1 leader_uuid: "db28e04bd78c460284a5f1561c6bb912" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } health_report { overall_health: UNKNOWN } } }
I20260807 02:25:17.559168 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.555357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39352 (local address 127.4.221.66:39553)
0807 02:25:17.555495 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:17.555501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:17.555545 (+ 44us) server_negotiation.cc:407] Connection header received
0807 02:25:17.555759 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:17.555762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:17.555833 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:17.555948 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:25:17.556204 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.556724 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.557571 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:17.557809 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:17.557967 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:17.558323 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:25:17.558742 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:25:17.558873 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:17.558937 (+ 64us) server_negotiation.cc:299] Negotiation successful
0807 02:25:17.558997 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 02:25:17.565289 21923 raft_consensus.cc:1261] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37 [term 1 FOLLOWER]: Refusing update from remote peer db28e04bd78c460284a5f1561c6bb912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:25:17.565265 21642 raft_consensus.cc:1261] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Refusing update from remote peer db28e04bd78c460284a5f1561c6bb912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:25:17.566089 21989 consensus_queue.cc:1035] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:25:17.566330 21991 consensus_queue.cc:1035] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:25:17.573061 21993 mvcc.cc:205] Tried to move back new op lower bound from 7315740743939694592 to 7315740743853600768. Current Snapshot: MvccSnapshot[applied={T|T < 7315740743939694592}]
I20260807 02:25:17.573509 21994 mvcc.cc:205] Tried to move back new op lower bound from 7315740743939694592 to 7315740743853600768. Current Snapshot: MvccSnapshot[applied={T|T < 7315740743939694592}]
I20260807 02:25:17.582641 21491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:25:17.582870 21491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:25:17.585542 21491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Sending DeleteTablet for 3 replicas of tablet 4424f2bc44bc46c89a23a81846afc90a
I20260807 02:25:17.586324 21621 tablet_service.cc:1514] Processing DeleteTablet for tablet 4424f2bc44bc46c89a23a81846afc90a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:25:17 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:56320
I20260807 02:25:17.586301 21763 tablet_service.cc:1514] Processing DeleteTablet for tablet 4424f2bc44bc46c89a23a81846afc90a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:25:17 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:39340
I20260807 02:25:17.586421 21901 tablet_service.cc:1514] Processing DeleteTablet for tablet 4424f2bc44bc46c89a23a81846afc90a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:25:17 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:37752
I20260807 02:25:17.586978 22007 tablet_replica.cc:330] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37: stopping tablet replica
I20260807 02:25:17.587316 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 21532
I20260807 02:25:17.588979 22006 tablet_replica.cc:330] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912: stopping tablet replica
I20260807 02:25:17.589357 22006 raft_consensus.cc:2229] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:25:17.589977 22006 raft_consensus.cc:2258] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:25:17.587391 22007 raft_consensus.cc:2229] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:25:17.590972 22007 raft_consensus.cc:2258] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:25:17.592701 22007 ts_tablet_manager.cc:1905] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:25:17.595278 22006 ts_tablet_manager.cc:1905] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:25:17.595611 22007 ts_tablet_manager.cc:1918] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:25:17.595767 22007 log.cc:1196] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/4424f2bc44bc46c89a23a81846afc90a
I20260807 02:25:17.596161 22007 ts_tablet_manager.cc:1939] T 4424f2bc44bc46c89a23a81846afc90a P c316b081e1494e54a9c8b520526e4b37: Deleting consensus metadata
W20260807 02:25:17.597491 21473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:44017 (error 108)
I20260807 02:25:17.597591 21474 catalog_manager.cc:4841] TS c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137): tablet 4424f2bc44bc46c89a23a81846afc90a (table test-table [id=e52e6b002c6d47ac897b1c29ab06449e]) successfully deleted
I20260807 02:25:17.598901 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 21694
I20260807 02:25:17.600247 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.599260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:44017 (local address 127.0.0.1:56336)
0807 02:25:17.600052 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:17.600137 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:44017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":676}
W20260807 02:25:17.601198 21473 catalog_manager.cc:4584] TS 8c5489a1ae504bc5b0466e9652c1cb0a (127.4.221.65:44017): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4424f2bc44bc46c89a23a81846afc90a: Network error: Client connection negotiation failed: client connection to 127.4.221.65:44017: connect: Connection refused (error 111)
I20260807 02:25:17.606596 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 21835
I20260807 02:25:17.607880 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:17.606802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:39553 (local address 127.0.0.1:39364)
0807 02:25:17.607686 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:17.607761 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.66:39553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":770}
I20260807 02:25:17.612968 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 21456
I20260807 02:25:37.621364 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:45977
--webserver_interface=127.4.221.126
--webserver_port=43253
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:45977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:25:37.754086 22009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:37.754499 22009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:37.754582 22009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:37.758857 22009 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260807 02:25:37.758952 22009 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260807 02:25:37.758980 22009 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260807 02:25:37.759006 22009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:25:37.759066 22009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:37.759126 22009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:37.759168 22009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:25:37.759207 22009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:25:37.765007 22009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:45977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:45977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=43253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:37.766521 22009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:37.767818 22009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:37.778240 22014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:37.778232 22015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:37.778224 22017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:37.778292 22009 server_base.cc:1034] running on GCE node
I20260807 02:25:37.779229 22009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:37.780580 22009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:37.781757 22009 hybrid_clock.cc:648] HybridClock initialized: now 1786069537781738 us; error 36 us; skew 500 ppm
Aug 07 02:25:37 dist-test-slave-69rg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069537, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:37.791615 22009 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:25:37.793287 22009 webserver.cc:460] Webserver started at http://127.4.221.126:43253/ using document root <none> and password file <none>
I20260807 02:25:37.794272 22009 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:37.794365 22009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:37.799844 22009 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:25:37.802577 22025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:37.804109 22009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:25:37.804255 22009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "f56a6067d16f482a90c0d0c5f75382f6"
format_stamp: "Formatted at 2026-08-07 02:24:55 on dist-test-slave-69rg"
server_key: "93e2e995ff605092a9af472331b22272"
server_key_iv: "484c7ef14169b8983b753471dffbc885"
server_key_version: "encryptionkey@0"
I20260807 02:25:37.804992 22009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:37.834935 22009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:37.839226 22009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:37.839501 22009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:37.848888 22009 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:45977
I20260807 02:25:37.849238 22080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:45977 every 8 connection(s)
I20260807 02:25:37.850364 22009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:25:37.851405 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 22009
I20260807 02:25:37.851935 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:44017
--local_ip_for_outbound_sockets=127.4.221.65
--tserver_master_addrs=127.4.221.126:45977
--webserver_port=34421
--webserver_interface=127.4.221.65
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:25:37.856644 22081 sys_catalog.cc:263] Verifying existing consensus state
I20260807 02:25:37.858938 22081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Bootstrap starting.
I20260807 02:25:37.881273 22081 log.cc:824] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:37.894045 22081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:25:37.894536 22081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: Bootstrap complete.
I20260807 02:25:37.899291 22081 raft_consensus.cc:348] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } }
I20260807 02:25:37.899763 22081 raft_consensus.cc:729] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56a6067d16f482a90c0d0c5f75382f6, State: Initialized, Role: FOLLOWER
I20260807 02:25:37.900378 22081 consensus_queue.cc:260] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } }
I20260807 02:25:37.900543 22081 raft_consensus.cc:388] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:25:37.900636 22081 raft_consensus.cc:482] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:25:37.900767 22081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 2 FOLLOWER]: Advancing to term 3
I20260807 02:25:37.903045 22081 raft_consensus.cc:504] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } }
I20260807 02:25:37.903482 22081 leader_election.cc:302] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f56a6067d16f482a90c0d0c5f75382f6; no voters:
I20260807 02:25:37.903940 22081 leader_election.cc:288] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260807 02:25:37.904040 22085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 3 FOLLOWER]: Leader election won for term 3
I20260807 02:25:37.904274 22085 raft_consensus.cc:686] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [term 3 LEADER]: Becoming Leader. State: Replica: f56a6067d16f482a90c0d0c5f75382f6, State: Running, Role: LEADER
I20260807 02:25:37.904690 22085 consensus_queue.cc:237] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } }
I20260807 02:25:37.905441 22081 sys_catalog.cc:564] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:25:37.906366 22087 sys_catalog.cc:455] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f56a6067d16f482a90c0d0c5f75382f6. Latest consensus state: current_term: 3 leader_uuid: "f56a6067d16f482a90c0d0c5f75382f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } } }
I20260807 02:25:37.906409 22086 sys_catalog.cc:455] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f56a6067d16f482a90c0d0c5f75382f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a6067d16f482a90c0d0c5f75382f6" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 45977 } } }
I20260807 02:25:37.906503 22087 sys_catalog.cc:458] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [sys.catalog]: This master's current role is: LEADER
I20260807 02:25:37.906541 22086 sys_catalog.cc:458] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6 [sys.catalog]: This master's current role is: LEADER
I20260807 02:25:37.911566 22100 catalog_manager.cc:1259] Loaded cluster ID: d385d383e608403193d5e65db8fee151
I20260807 02:25:37.911654 22100 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: loading cluster ID for follower catalog manager: success
I20260807 02:25:37.913735 22100 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f56a6067d16f482a90c0d0c5f75382f6: acquiring CA information for follower catalog manager: success
F20260807 02:25:37.914908 22100 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7f56d999ad google::LogMessage::Fail() at ??:0
@ 0x7f7f56d9c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7f56d994de google::LogMessage::Flush() at ??:0
@ 0x7f7f56d9aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7f59a5d18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7f616fe989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f7f616db023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f7f616db54b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f7f616d4652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f7f616d39e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f7f6171a541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7f58fdc512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7f5a6206db start_thread at ??:0
@ 0x7f7f5589761f clone at ??:0
W20260807 02:25:38.020702 22083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:38.021108 22083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:38.021169 22083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:38.026401 22083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:38.026506 22083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:38.026624 22083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:25:38.033511 22083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:44017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=34421
--tserver_master_addrs=127.4.221.126:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:38.035189 22083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:38.036653 22083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:38.050395 22121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:38.050632 22122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:38.050402 22124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:38.050978 22083 server_base.cc:1034] running on GCE node
I20260807 02:25:38.051602 22083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:38.052623 22083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:38.053850 22083 hybrid_clock.cc:648] HybridClock initialized: now 1786069538053821 us; error 58 us; skew 500 ppm
Aug 07 02:25:38 dist-test-slave-69rg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069538, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:38.058284 22083 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:25:38.059885 22083 webserver.cc:460] Webserver started at http://127.4.221.65:34421/ using document root <none> and password file <none>
I20260807 02:25:38.060932 22083 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:38.061061 22083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:38.068974 22083 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260807 02:25:38.072660 22131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:38.074585 22083 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260807 02:25:38.074752 22083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "8c5489a1ae504bc5b0466e9652c1cb0a"
format_stamp: "Formatted at 2026-08-07 02:24:55 on dist-test-slave-69rg"
server_key: "cca6f77050a51fd3a77688a33d438748"
server_key_iv: "8536e69d0f6a730b4c95ad29b574b459"
server_key_version: "encryptionkey@0"
I20260807 02:25:38.075505 22083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:38.109433 22083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:38.113896 22083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:38.114161 22083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:38.114977 22083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:25:38.116437 22138 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260807 02:25:38.123242 22083 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260807 02:25:38.123359 22083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260807 02:25:38.123476 22083 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260807 02:25:38.125736 22083 ts_tablet_manager.cc:610] Registered 1 tablets
I20260807 02:25:38.125826 22083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260807 02:25:38.126245 22138 tablet_bootstrap.cc:492] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a: Bootstrap starting.
I20260807 02:25:38.139014 22083 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:44017
I20260807 02:25:38.139297 22252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:44017 every 8 connection(s)
I20260807 02:25:38.140339 22083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:25:38.143445 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 22083
I20260807 02:25:38.144086 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:39553
--local_ip_for_outbound_sockets=127.4.221.66
--tserver_master_addrs=127.4.221.126:45977
--webserver_port=37791
--webserver_interface=127.4.221.66
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:25:38.155850 22138 log.cc:824] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a: Log is configured to *not* fsync() on all Append() calls
W20260807 02:25:38.159785 22255 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:25:38.142940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:52703)
0807 02:25:38.143674 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:38.143746 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:38.145247 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:38.158737 (+ 13490us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":170,"threads_started":1}
W20260807 02:25:38.160497 22253 heartbeater.cc:644] Failed to heartbeat to 127.4.221.126:45977 (0 consecutive failures): Network error: Failed to ping master at 127.4.221.126:45977: Client connection negotiation failed: client connection to 127.4.221.126:45977: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 02:25:38.161388 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:38.161022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:54797)
0807 02:25:38.161170 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:38.161301 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 02:25:38.162196 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:38.161972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:34749)
0807 02:25:38.162082 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:38.162138 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 02:25:38.162436 22253 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 02:25:38.167474 22138 tablet_bootstrap.cc:492] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260807 02:25:38.168056 22138 tablet_bootstrap.cc:492] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a: Bootstrap complete.
I20260807 02:25:38.170027 22138 ts_tablet_manager.cc:1397] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a: Time spent bootstrapping tablet: real 0.044s user 0.034s sys 0.007s
I20260807 02:25:38.173995 22138 raft_consensus.cc:348] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:38.174675 22138 raft_consensus.cc:729] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c5489a1ae504bc5b0466e9652c1cb0a, State: Initialized, Role: FOLLOWER
I20260807 02:25:38.175328 22138 consensus_queue.cc:260] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:38.176617 22138 ts_tablet_manager.cc:1428] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 02:25:38.176836 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:38.176511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:53165)
0807 02:25:38.176656 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:38.176737 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260807 02:25:38.177603 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:38.177360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:34391)
0807 02:25:38.177488 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:38.177541 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260807 02:25:38.288128 22256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:38.288470 22256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:38.288520 22256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:38.293044 22256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:38.293128 22256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:38.293231 22256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:25:38.299078 22256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:39553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=37791
--tserver_master_addrs=127.4.221.126:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:38.300500 22256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:38.301837 22256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:38.313905 22263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:38.313915 22266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:38.313920 22264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:38.314213 22256 server_base.cc:1034] running on GCE node
I20260807 02:25:38.314745 22256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:38.315493 22256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:38.316694 22256 hybrid_clock.cc:648] HybridClock initialized: now 1786069538316661 us; error 44 us; skew 500 ppm
Aug 07 02:25:38 dist-test-slave-69rg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069538, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:38.320319 22256 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:25:38.321485 22256 webserver.cc:460] Webserver started at http://127.4.221.66:37791/ using document root <none> and password file <none>
I20260807 02:25:38.322297 22256 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:38.322389 22256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:38.327626 22256 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260807 02:25:38.330312 22273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:38.331884 22256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:25:38.332055 22256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "db28e04bd78c460284a5f1561c6bb912"
format_stamp: "Formatted at 2026-08-07 02:24:55 on dist-test-slave-69rg"
server_key: "f0da3b3baebbeaf8963a1948416145bd"
server_key_iv: "c44208ff5c2649179752e03534af1a4b"
server_key_version: "encryptionkey@0"
I20260807 02:25:38.332799 22256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:38.354589 22256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:38.359098 22256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:38.359372 22256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:38.360216 22256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:25:38.361660 22280 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260807 02:25:38.366901 22280 ts_tablet_manager.cc:1852] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260807 02:25:38.367439 22280 ts_tablet_manager.cc:1905] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:25:38.372336 22280 ts_tablet_manager.cc:1918] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:25:38.372545 22280 log.cc:1196] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/4424f2bc44bc46c89a23a81846afc90a
I20260807 02:25:38.372954 22280 ts_tablet_manager.cc:1939] T 4424f2bc44bc46c89a23a81846afc90a P db28e04bd78c460284a5f1561c6bb912: Deleting consensus metadata
I20260807 02:25:38.373329 22256 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260807 02:25:38.373397 22256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.000s sys 0.000s
I20260807 02:25:38.373687 22256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:25:38.373718 22256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:38.385859 22256 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:39553
I20260807 02:25:38.385931 22393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:39553 every 8 connection(s)
I20260807 02:25:38.387009 22256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:25:38.391547 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:38.389758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:55177)
0807 02:25:38.390532 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:38.390670 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":176,"threads_started":1}
W20260807 02:25:38.392107 22394 heartbeater.cc:644] Failed to heartbeat to 127.4.221.126:45977 (0 consecutive failures): Network error: Failed to ping master at 127.4.221.126:45977: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
I20260807 02:25:38.392992 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:38.392678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:45907)
0807 02:25:38.392828 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:38.392918 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 02:25:38.393903 22396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:38.393555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:59193)
0807 02:25:38.393713 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:38.393775 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260807 02:25:38.394116 22394 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 02:25:38.396554 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 22256
I20260807 02:25:38.397142 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:35137
--local_ip_for_outbound_sockets=127.4.221.67
--tserver_master_addrs=127.4.221.126:45977
--webserver_port=40017
--webserver_interface=127.4.221.67
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:25:38.532276 22397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:38.532649 22397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:38.532703 22397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:38.537240 22397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:38.537335 22397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:38.537446 22397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:25:38.543574 22397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:35137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=40017
--tserver_master_addrs=127.4.221.126:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:38.545104 22397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:38.546522 22397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:38.558270 22405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:38.558159 22403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:38.558169 22402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:38.558594 22397 server_base.cc:1034] running on GCE node
I20260807 02:25:38.559188 22397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:38.559983 22397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:38.561208 22397 hybrid_clock.cc:648] HybridClock initialized: now 1786069538561189 us; error 30 us; skew 500 ppm
Aug 07 02:25:38 dist-test-slave-69rg krb5kdc[20873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069538, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:38.565091 22397 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:25:38.566430 22397 webserver.cc:460] Webserver started at http://127.4.221.67:40017/ using document root <none> and password file <none>
I20260807 02:25:38.567162 22397 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:38.567235 22397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:38.572343 22397 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 02:25:38.574997 22412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:38.576565 22397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:25:38.576715 22397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "c316b081e1494e54a9c8b520526e4b37"
format_stamp: "Formatted at 2026-08-07 02:24:56 on dist-test-slave-69rg"
server_key: "e28b8dda8bf7caa0f89352e62f3734aa"
server_key_iv: "0c8487685afa346a61745d6affa13ff4"
server_key_version: "encryptionkey@0"
I20260807 02:25:38.577423 22397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:38.596755 22397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:38.601432 22397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:38.601734 22397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:38.602604 22397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:25:38.603680 22397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:25:38.603736 22397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:38.603814 22397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:25:38.603845 22397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:38.614269 22397 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:35137
I20260807 02:25:38.614288 22530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:35137 every 8 connection(s)
I20260807 02:25:38.615484 22397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:25:38.617739 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 22397
I20260807 02:25:38.620327 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:38.618198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:34291)
0807 02:25:38.618870 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:38.619443 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":158,"threads_started":1}
W20260807 02:25:38.620826 22531 heartbeater.cc:644] Failed to heartbeat to 127.4.221.126:45977 (0 consecutive failures): Network error: Failed to ping master at 127.4.221.126:45977: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
I20260807 02:25:38.621510 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:38.621256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:50217)
0807 02:25:38.621389 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:38.621448 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260807 02:25:38.622200 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:38.621981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:42633)
0807 02:25:38.622086 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:38.622142 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 02:25:38.622364 22531 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260807 02:25:39.179026 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:39.178424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:45549)
0807 02:25:39.178779 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:39.178846 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":89,"threads_started":1}
I20260807 02:25:39.395242 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:39.394783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:53045)
0807 02:25:39.395066 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:39.395137 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260807 02:25:39.623939 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:39.623227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:40679)
0807 02:25:39.623691 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:39.623771 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":179,"threads_started":1}
I20260807 02:25:39.885989 22539 raft_consensus.cc:482] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:25:39.886365 22539 raft_consensus.cc:504] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:39.887784 22539 leader_election.cc:288] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db28e04bd78c460284a5f1561c6bb912 (127.4.221.66:39553), c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137)
Aug 07 02:25:39 dist-test-slave-69rg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069538, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.66@KRBTEST.COM
Aug 07 02:25:39 dist-test-slave-69rg krb5kdc[20873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069538, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.67@KRBTEST.COM
I20260807 02:25:39.905949 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:39.888283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:46185 (local address 127.4.221.66:39553)
0807 02:25:39.888680 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:39.888702 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:39.888760 (+ 58us) server_negotiation.cc:407] Connection header received
0807 02:25:39.890011 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:39.890025 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:39.890604 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:39.891010 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:39.893124 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:39.894873 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:39.896293 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:39.896881 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:39.897068 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:39.900688 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:39.900694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:39.900709 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:39.900742 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:39.902902 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:39.903833 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:39.903838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:39.903841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:39.903969 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:39.904278 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:39.904284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:39.904288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:39.904671 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:39.904810 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:39.905363 (+ 553us) server_negotiation.cc:299] Negotiation successful
0807 02:25:39.905678 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":176,"threads_started":1}
I20260807 02:25:39.906141 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:39.888915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:45265 (local address 127.4.221.67:35137)
0807 02:25:39.889286 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:39.889309 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:39.889349 (+ 40us) server_negotiation.cc:407] Connection header received
0807 02:25:39.890314 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:39.890321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:39.890624 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:39.891010 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:39.893124 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:39.894872 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:39.896320 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:39.896836 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:39.897035 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:39.900783 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:39.900787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:39.900798 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:39.900829 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:39.902779 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:39.903585 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:39.903590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:39.903594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:39.903714 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:39.904067 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:39.904071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:39.904073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:39.904592 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:39.904766 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:39.905601 (+ 835us) server_negotiation.cc:299] Negotiation successful
0807 02:25:39.905898 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":116,"threads_started":1}
I20260807 02:25:39.906169 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:39.888571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:35137 (local address 127.4.221.65:45265)
0807 02:25:39.888853 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:39.888868 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:39.888985 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:39.891131 (+ 2146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:39.891139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:39.891913 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:25:39.892881 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:39.892907 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:39.895078 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:39.895081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:39.896161 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:39.896172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:39.896977 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:39.896980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:39.897149 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:39.897960 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:25:39.897992 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:25:39.900585 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:25:39.902917 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:39.902924 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:39.902938 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:39.903464 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:39.903816 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:39.903818 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:39.903819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:39.903964 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:39.904726 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:25:39.904733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:25:39.905019 (+ 286us) client_negotiation.cc:769] Sending connection context
0807 02:25:39.905245 (+ 226us) client_negotiation.cc:240] Negotiation successful
0807 02:25:39.905967 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":622,"spinlock_wait_cycles":3712,"thread_start_us":67,"threads_started":1}
I20260807 02:25:39.906373 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:39.888099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:39553 (local address 127.4.221.65:46185)
0807 02:25:39.888383 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:39.888400 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:39.888541 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:39.891131 (+ 2590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:39.891139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:39.891918 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:25:39.892881 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:39.892907 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:39.895072 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:39.895075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:39.896157 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:39.896170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:39.897259 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:39.897262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:39.897349 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:39.898042 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:25:39.898053 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:25:39.900364 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:25:39.903008 (+ 2644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:39.903012 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:39.903014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:39.903464 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:39.904085 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:39.904088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:39.904090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:39.904182 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:39.904876 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:25:39.904881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:25:39.905023 (+ 142us) client_negotiation.cc:769] Sending connection context
0807 02:25:39.905245 (+ 222us) client_negotiation.cc:240] Negotiation successful
0807 02:25:39.905690 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":231,"spinlock_wait_cycles":47744,"thread_start_us":110,"threads_started":1}
W20260807 02:25:39.909849 22133 leader_election.cc:341] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137): Not found: Tablet not found: 4424f2bc44bc46c89a23a81846afc90a
W20260807 02:25:39.910516 22133 leader_election.cc:341] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db28e04bd78c460284a5f1561c6bb912 (127.4.221.66:39553): Not found: Tablet not found: 4424f2bc44bc46c89a23a81846afc90a
I20260807 02:25:39.910610 22133 leader_election.cc:302] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c5489a1ae504bc5b0466e9652c1cb0a; no voters: c316b081e1494e54a9c8b520526e4b37, db28e04bd78c460284a5f1561c6bb912
I20260807 02:25:39.910885 22539 raft_consensus.cc:2726] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 02:25:40.180545 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:40.180123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:55979)
0807 02:25:40.180300 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:40.180396 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260807 02:25:40.396725 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:40.396184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:35365)
0807 02:25:40.396543 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:40.396609 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":128,"threads_started":1}
I20260807 02:25:40.625749 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:40.625113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:50187)
0807 02:25:40.625512 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:40.625597 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":135,"threads_started":1}
I20260807 02:25:41.182379 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:41.181641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:40091)
0807 02:25:41.182160 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:41.182240 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":189,"threads_started":1}
I20260807 02:25:41.398449 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:41.397903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:43791)
0807 02:25:41.398250 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:41.398320 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":107,"threads_started":1}
I20260807 02:25:41.627368 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:41.626841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:33489)
0807 02:25:41.627152 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:41.627222 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":106,"threads_started":1}
I20260807 02:25:41.667508 22550 raft_consensus.cc:482] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:25:41.667677 22550 raft_consensus.cc:504] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:41.668269 22550 leader_election.cc:288] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db28e04bd78c460284a5f1561c6bb912 (127.4.221.66:39553), c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137)
W20260807 02:25:41.669428 22133 leader_election.cc:341] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db28e04bd78c460284a5f1561c6bb912 (127.4.221.66:39553): Not found: Tablet not found: 4424f2bc44bc46c89a23a81846afc90a
W20260807 02:25:41.669631 22133 leader_election.cc:341] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137): Not found: Tablet not found: 4424f2bc44bc46c89a23a81846afc90a
I20260807 02:25:41.669809 22133 leader_election.cc:302] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c5489a1ae504bc5b0466e9652c1cb0a; no voters: c316b081e1494e54a9c8b520526e4b37, db28e04bd78c460284a5f1561c6bb912
I20260807 02:25:41.670083 22550 raft_consensus.cc:2726] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 02:25:42.184190 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:42.183605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:49353)
0807 02:25:42.183972 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:42.184052 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
I20260807 02:25:42.400413 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:42.399704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:59339)
0807 02:25:42.400175 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:42.400277 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":125,"threads_started":1}
I20260807 02:25:42.628858 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:42.628305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:44353)
0807 02:25:42.628649 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:42.628726 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
I20260807 02:25:43.186355 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:43.185647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:33761)
0807 02:25:43.186092 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:43.186238 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":99,"threads_started":1}
I20260807 02:25:43.364711 22555 raft_consensus.cc:482] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:25:43.364912 22555 raft_consensus.cc:504] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:43.365520 22555 leader_election.cc:288] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db28e04bd78c460284a5f1561c6bb912 (127.4.221.66:39553), c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137)
W20260807 02:25:43.366528 22133 leader_election.cc:341] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137): Not found: Tablet not found: 4424f2bc44bc46c89a23a81846afc90a
W20260807 02:25:43.366683 22133 leader_election.cc:341] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db28e04bd78c460284a5f1561c6bb912 (127.4.221.66:39553): Not found: Tablet not found: 4424f2bc44bc46c89a23a81846afc90a
I20260807 02:25:43.366739 22133 leader_election.cc:302] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c5489a1ae504bc5b0466e9652c1cb0a; no voters: c316b081e1494e54a9c8b520526e4b37, db28e04bd78c460284a5f1561c6bb912
I20260807 02:25:43.366884 22555 raft_consensus.cc:2726] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 02:25:43.401772 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:43.401327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:44739)
0807 02:25:43.401593 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:43.401655 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260807 02:25:43.630522 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:43.630009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:41321)
0807 02:25:43.630321 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:43.630393 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":89,"threads_started":1}
I20260807 02:25:44.187595 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:44.187119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:37589)
0807 02:25:44.187422 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:44.187497 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":103,"threads_started":1}
I20260807 02:25:44.403119 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:44.402640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:41823)
0807 02:25:44.402941 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:44.403015 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":99,"threads_started":1}
I20260807 02:25:44.631745 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:44.631303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:34117)
0807 02:25:44.631584 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:44.631650 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":123,"threads_started":1}
I20260807 02:25:45.189440 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:45.188717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:44147)
0807 02:25:45.189144 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:45.189235 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":128,"threads_started":1}
I20260807 02:25:45.404923 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:45.404206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:39601)
0807 02:25:45.404696 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:45.404775 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":146,"threads_started":1}
I20260807 02:25:45.633240 22563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:45.632663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:37903)
0807 02:25:45.633044 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:45.633110 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":100,"threads_started":1}
I20260807 02:25:45.977181 22564 raft_consensus.cc:482] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:25:45.977367 22564 raft_consensus.cc:504] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:45.977982 22564 leader_election.cc:288] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db28e04bd78c460284a5f1561c6bb912 (127.4.221.66:39553), c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137)
W20260807 02:25:45.978995 22133 leader_election.cc:341] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db28e04bd78c460284a5f1561c6bb912 (127.4.221.66:39553): Not found: Tablet not found: 4424f2bc44bc46c89a23a81846afc90a
W20260807 02:25:45.979166 22133 leader_election.cc:341] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137): Not found: Tablet not found: 4424f2bc44bc46c89a23a81846afc90a
I20260807 02:25:45.979226 22133 leader_election.cc:302] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c5489a1ae504bc5b0466e9652c1cb0a; no voters: c316b081e1494e54a9c8b520526e4b37, db28e04bd78c460284a5f1561c6bb912
I20260807 02:25:45.979439 22564 raft_consensus.cc:2726] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 02:25:46.191110 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:46.190556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:35543)
0807 02:25:46.190916 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:46.190983 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":162,"threads_started":1}
I20260807 02:25:46.406788 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:46.406182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:56751)
0807 02:25:46.406570 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:46.406655 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":125,"threads_started":1}
I20260807 02:25:46.634861 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:46.634314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:53641)
0807 02:25:46.634637 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:46.634721 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":118,"threads_started":1}
I20260807 02:25:47.192853 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:47.192170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:32823)
0807 02:25:47.192607 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:47.192696 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":118,"threads_started":1}
I20260807 02:25:47.408268 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:47.407765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:34801)
0807 02:25:47.408075 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:47.408140 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":108,"threads_started":1}
I20260807 02:25:47.636349 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:47.635798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:54559)
0807 02:25:47.636158 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:47.636225 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":123,"threads_started":1}
I20260807 02:25:48.161705 22571 raft_consensus.cc:482] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:25:48.161953 22571 raft_consensus.cc:504] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db28e04bd78c460284a5f1561c6bb912" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 39553 } } peers { permanent_uuid: "c316b081e1494e54a9c8b520526e4b37" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 35137 } } peers { permanent_uuid: "8c5489a1ae504bc5b0466e9652c1cb0a" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 44017 } }
I20260807 02:25:48.162556 22571 leader_election.cc:288] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers db28e04bd78c460284a5f1561c6bb912 (127.4.221.66:39553), c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137)
W20260807 02:25:48.163496 22133 leader_election.cc:341] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c316b081e1494e54a9c8b520526e4b37 (127.4.221.67:35137): Not found: Tablet not found: 4424f2bc44bc46c89a23a81846afc90a
W20260807 02:25:48.163641 22133 leader_election.cc:341] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer db28e04bd78c460284a5f1561c6bb912 (127.4.221.66:39553): Not found: Tablet not found: 4424f2bc44bc46c89a23a81846afc90a
I20260807 02:25:48.163694 22133 leader_election.cc:302] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c5489a1ae504bc5b0466e9652c1cb0a; no voters: c316b081e1494e54a9c8b520526e4b37, db28e04bd78c460284a5f1561c6bb912
I20260807 02:25:48.163856 22571 raft_consensus.cc:2726] T 4424f2bc44bc46c89a23a81846afc90a P 8c5489a1ae504bc5b0466e9652c1cb0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260807 02:25:48.194578 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:48.194073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:49227)
0807 02:25:48.194379 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:48.194452 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
I20260807 02:25:48.410006 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:48.409415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:52689)
0807 02:25:48.409758 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:48.409867 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":139,"threads_started":1}
I20260807 02:25:48.638000 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:48.637339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:49779)
0807 02:25:48.637709 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:48.637781 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":128,"threads_started":1}
I20260807 02:25:49.196014 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:49.195500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:43943)
0807 02:25:49.195813 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:49.195882 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":111,"threads_started":1}
I20260807 02:25:49.411664 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:49.411021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:54483)
0807 02:25:49.411440 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:49.411521 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":157,"threads_started":1}
I20260807 02:25:49.639421 22579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:49.638916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:34969)
0807 02:25:49.639231 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:49.639297 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":119,"threads_started":1}
I20260807 02:25:50.197808 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:50.197166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:38717)
0807 02:25:50.197597 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:50.197678 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":131,"threads_started":1}
I20260807 02:25:50.413296 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:50.412718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:46915)
0807 02:25:50.413092 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:50.413166 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":130,"threads_started":1}
I20260807 02:25:50.641098 22582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:50.640522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:44461)
0807 02:25:50.640894 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:50.640970 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":130,"threads_started":1}
I20260807 02:25:51.199672 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:51.198939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:34755)
0807 02:25:51.199385 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:51.199478 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":186,"threads_started":1}
I20260807 02:25:51.414860 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:51.414340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:34349)
0807 02:25:51.414660 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:51.414732 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260807 02:25:51.642575 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:51.642098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:35397)
0807 02:25:51.642370 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:51.642430 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
I20260807 02:25:52.201136 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:52.200599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:57163)
0807 02:25:52.200937 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:52.201008 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":102,"threads_started":1}
I20260807 02:25:52.416616 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:52.416010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:45615)
0807 02:25:52.416402 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:52.416481 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":114,"threads_started":1}
I20260807 02:25:52.644042 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:52.643479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.67:37285)
0807 02:25:52.643839 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:52.643912 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":132,"threads_started":1}
I20260807 02:25:53.202911 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:53.202414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.65:56555)
0807 02:25:53.202718 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:53.202785 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":100,"threads_started":1}
I20260807 02:25:53.418434 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:53.417786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:45977 (local address 127.4.221.66:34113)
0807 02:25:53.418238 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:53.418308 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.126:45977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":133,"threads_started":1}
I20260807 02:25:53.620217 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 22083
I20260807 02:25:53.629048 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 22256
I20260807 02:25:53.637631 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 22397
2026-08-07T02:25:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59684 ms)
[----------] 30 tests from SecurityITest (251792 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:25:53 dist-test-slave-69rg krb5kdc[22594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:25:53 dist-test-slave-69rg krb5kdc[22594](info): set up 2 sockets
Aug 07 02:25:53 dist-test-slave-69rg krb5kdc[22594](info): commencing operation
krb5kdc: starting...
W20260807 02:25:54.718273 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:25:54 dist-test-slave-69rg krb5kdc[22594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:25:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:25:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:25:54.845552 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:40161
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:40161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:25:54.965167 22607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:54.965410 22607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:54.965449 22607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:54.969313 22607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:25:54.969367 22607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:54.969385 22607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:54.969401 22607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:25:54.969415 22607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:25:54.974473 22607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:40161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:40161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:54.975404 22607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:54.976339 22607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:54.985420 22616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:54.985406 22612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:54.985565 22613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:54.985464 22607 server_base.cc:1034] running on GCE node
I20260807 02:25:54.986191 22607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:54.987288 22607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:54.988451 22607 hybrid_clock.cc:648] HybridClock initialized: now 1786069554988427 us; error 41 us; skew 500 ppm
Aug 07 02:25:54 dist-test-slave-69rg krb5kdc[22594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069554, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:54.991206 22607 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:25:54.992121 22607 webserver.cc:460] Webserver started at http://127.4.221.126:42835/ using document root <none> and password file <none>
I20260807 02:25:54.992693 22607 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:54.992784 22607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:54.993013 22607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:25:54.995081 22607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "af9e094a3b0544dc96aa79b36a31bb88"
format_stamp: "Formatted at 2026-08-07 02:25:54 on dist-test-slave-69rg"
server_key: "da68fc077b3ff7a8a3ff0ad6de5c968d"
server_key_iv: "825143297eb6be462f9137221c429aa8"
server_key_version: "encryptionkey@0"
I20260807 02:25:54.995671 22607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "af9e094a3b0544dc96aa79b36a31bb88"
format_stamp: "Formatted at 2026-08-07 02:25:54 on dist-test-slave-69rg"
server_key: "da68fc077b3ff7a8a3ff0ad6de5c968d"
server_key_iv: "825143297eb6be462f9137221c429aa8"
server_key_version: "encryptionkey@0"
I20260807 02:25:54.999410 22607 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 02:25:55.001905 22623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:55.003178 22607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:25:55.003333 22607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "af9e094a3b0544dc96aa79b36a31bb88"
format_stamp: "Formatted at 2026-08-07 02:25:54 on dist-test-slave-69rg"
server_key: "da68fc077b3ff7a8a3ff0ad6de5c968d"
server_key_iv: "825143297eb6be462f9137221c429aa8"
server_key_version: "encryptionkey@0"
I20260807 02:25:55.003446 22607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:55.020004 22607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:55.023334 22607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:55.023573 22607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:55.031466 22607 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:40161
I20260807 02:25:55.031499 22676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:40161 every 8 connection(s)
I20260807 02:25:55.032507 22607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:25:55.035667 22677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:25:55.041277 22677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88: Bootstrap starting.
I20260807 02:25:55.042747 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 22607
I20260807 02:25:55.042940 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:25:55.043247 4981 external_mini_cluster.cc:1428] Setting key f042d62d5115dd8289d520fcf476bca7
I20260807 02:25:55.044227 22677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88: Neither blocks nor log segments found. Creating new log.
I20260807 02:25:55.045152 22677 log.cc:824] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:55.047747 22677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88: No bootstrap required, opened a new log
I20260807 02:25:55.052498 22677 raft_consensus.cc:348] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9e094a3b0544dc96aa79b36a31bb88" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40161 } }
I20260807 02:25:55.052824 22677 raft_consensus.cc:374] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:25:55.052908 22677 raft_consensus.cc:729] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9e094a3b0544dc96aa79b36a31bb88, State: Initialized, Role: FOLLOWER
Aug 07 02:25:55 dist-test-slave-69rg krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:25:55.053489 22677 consensus_queue.cc:260] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9e094a3b0544dc96aa79b36a31bb88" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40161 } }
I20260807 02:25:55.053694 22677 raft_consensus.cc:388] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:25:55.053784 22677 raft_consensus.cc:482] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:25:55.053937 22677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:25:55.055269 22677 raft_consensus.cc:504] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9e094a3b0544dc96aa79b36a31bb88" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40161 } }
I20260807 02:25:55.055697 22677 leader_election.cc:302] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af9e094a3b0544dc96aa79b36a31bb88; no voters:
I20260807 02:25:55.056068 22677 leader_election.cc:288] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:25:55.056154 22682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:25:55.056361 22682 raft_consensus.cc:686] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [term 1 LEADER]: Becoming Leader. State: Replica: af9e094a3b0544dc96aa79b36a31bb88, State: Running, Role: LEADER
I20260807 02:25:55.056689 22682 consensus_queue.cc:237] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9e094a3b0544dc96aa79b36a31bb88" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40161 } }
I20260807 02:25:55.057510 22677 sys_catalog.cc:564] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:25:55.058568 22683 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af9e094a3b0544dc96aa79b36a31bb88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9e094a3b0544dc96aa79b36a31bb88" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40161 } } }
I20260807 02:25:55.058928 22683 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [sys.catalog]: This master's current role is: LEADER
I20260807 02:25:55.059804 22684 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af9e094a3b0544dc96aa79b36a31bb88. Latest consensus state: current_term: 1 leader_uuid: "af9e094a3b0544dc96aa79b36a31bb88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9e094a3b0544dc96aa79b36a31bb88" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40161 } } }
I20260807 02:25:55.060006 22684 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88 [sys.catalog]: This master's current role is: LEADER
I20260807 02:25:55.060146 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:55.044471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.4.221.126:40161)
0807 02:25:55.044958 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:55.044977 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:55.045023 (+ 46us) server_negotiation.cc:407] Connection header received
0807 02:25:55.045892 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:55.045911 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:55.046346 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:55.046779 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:55.047427 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.049065 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.050123 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.050978 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.051228 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:55.054211 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:55.054233 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:55.054246 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:55.054274 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:55.056153 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:55.056686 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:55.056692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:55.056697 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:55.056840 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:55.057159 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:55.057164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:55.057166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:55.057679 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:55.057929 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:55.058744 (+ 815us) server_negotiation.cc:299] Negotiation successful
0807 02:25:55.059110 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":143,"threads_started":1}
I20260807 02:25:55.060328 22692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:25:55.063714 22692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:25:55.069443 22692 catalog_manager.cc:1347] Generated new cluster ID: b82ef500cbc44bf285198fe773d97be5
I20260807 02:25:55.069514 22692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:25:55.101469 22692 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:25:55.102625 22692 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:25:55.118741 22692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88: Generated new TSK 0
I20260807 02:25:55.119503 22692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:25:55.154364 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40161
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:25:55.271308 22704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:55.271548 22704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:55.271591 22704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:55.275879 22704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:55.275939 22704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:55.276028 22704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:25:55.281347 22704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:55.282415 22704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:55.283417 22704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 02:25:55.293165 22704 server_base.cc:1034] running on GCE node
W20260807 02:25:55.293131 22709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:55.293318 22710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:55.293187 22713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:55.293743 22704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:55.294431 22704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:55.295609 22704 hybrid_clock.cc:648] HybridClock initialized: now 1786069555295590 us; error 30 us; skew 500 ppm
Aug 07 02:25:55 dist-test-slave-69rg krb5kdc[22594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069555, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:55.298571 22704 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:25:55.299445 22704 webserver.cc:460] Webserver started at http://127.4.221.65:37921/ using document root <none> and password file <none>
I20260807 02:25:55.299971 22704 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:55.300031 22704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:55.300194 22704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:25:55.302310 22704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "bd7e24ba08f744c8926d9727a9512592"
format_stamp: "Formatted at 2026-08-07 02:25:55 on dist-test-slave-69rg"
server_key: "87d401d6d481f1df125dba9155a5335c"
server_key_iv: "4558d615a1debcf24c8bdec09f177a8c"
server_key_version: "encryptionkey@0"
I20260807 02:25:55.302851 22704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "bd7e24ba08f744c8926d9727a9512592"
format_stamp: "Formatted at 2026-08-07 02:25:55 on dist-test-slave-69rg"
server_key: "87d401d6d481f1df125dba9155a5335c"
server_key_iv: "4558d615a1debcf24c8bdec09f177a8c"
server_key_version: "encryptionkey@0"
I20260807 02:25:55.306578 22704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 02:25:55.309015 22721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:55.310304 22704 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:25:55.310436 22704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "bd7e24ba08f744c8926d9727a9512592"
format_stamp: "Formatted at 2026-08-07 02:25:55 on dist-test-slave-69rg"
server_key: "87d401d6d481f1df125dba9155a5335c"
server_key_iv: "4558d615a1debcf24c8bdec09f177a8c"
server_key_version: "encryptionkey@0"
I20260807 02:25:55.310537 22704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:55.324807 22704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:55.328094 22704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:55.328300 22704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:55.328895 22704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:25:55.329706 22704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:25:55.329753 22704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:55.329821 22704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:25:55.329875 22704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:55.339345 22704 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:45337
I20260807 02:25:55.339368 22837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:45337 every 8 connection(s)
I20260807 02:25:55.340257 22704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:25:55.341749 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 22704
I20260807 02:25:55.341921 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:25:55.342293 4981 external_mini_cluster.cc:1428] Setting key adfe2bfcfeabdbf5387790bb7f8f1976
Aug 07 02:25:55 dist-test-slave-69rg krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069555, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
I20260807 02:25:55.353946 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:55.342105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:52271 (local address 127.4.221.126:40161)
0807 02:25:55.342283 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:55.342289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:55.343255 (+ 966us) server_negotiation.cc:407] Connection header received
0807 02:25:55.344361 (+ 1106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:55.344367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:55.344427 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:55.344519 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:55.345630 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.346260 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.347417 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.347673 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.347789 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:55.350259 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:55.350263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:55.350265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:55.350292 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:55.351747 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:55.352334 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:55.352337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:55.352338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:55.352413 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:55.352773 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:55.352777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:55.352779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:55.352973 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:55.353089 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:55.353558 (+ 469us) server_negotiation.cc:299] Negotiation successful
0807 02:25:55.353723 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
I20260807 02:25:55.354893 22840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:55.342538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40161 (local address 127.4.221.65:52271)
0807 02:25:55.343057 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:55.343135 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:55.344160 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:55.344662 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:55.344668 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:55.345011 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:25:55.345470 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:55.345483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.346396 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.346401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:55.347279 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:55.347289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.347830 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.347834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:55.348003 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:55.348609 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:25:55.348633 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:25:55.350098 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:25:55.351895 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:55.351901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:55.351915 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:55.352242 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:55.352498 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:55.352500 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:55.352502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:55.352669 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:55.353086 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:25:55.353092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:25:55.353334 (+ 242us) client_negotiation.cc:769] Sending connection context
0807 02:25:55.353516 (+ 182us) client_negotiation.cc:240] Negotiation successful
0807 02:25:55.353870 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":142,"threads_started":1}
I20260807 02:25:55.356282 22838 heartbeater.cc:347] Connected to a master server at 127.4.221.126:40161
I20260807 02:25:55.356554 22838 heartbeater.cc:464] Registering TS with master...
I20260807 02:25:55.357137 22838 heartbeater.cc:511] Master 127.4.221.126:40161 requested a full tablet report, sending...
I20260807 02:25:55.359076 22640 ts_manager.cc:194] Registered new tserver with Master: bd7e24ba08f744c8926d9727a9512592 (127.4.221.65:45337)
I20260807 02:25:55.360821 22640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:52271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:25:55.372347 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40161
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:25:55.482439 22843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:55.482686 22843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:55.482756 22843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:55.486722 22843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:55.486776 22843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:55.486884 22843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:25:55.491993 22843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:55.493010 22843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:55.494046 22843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:55.503504 22849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:55.503515 22850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:55.503515 22852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:55.503569 22843 server_base.cc:1034] running on GCE node
I20260807 02:25:55.504137 22843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:55.504756 22843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:55.505889 22843 hybrid_clock.cc:648] HybridClock initialized: now 1786069555505877 us; error 27 us; skew 500 ppm
Aug 07 02:25:55 dist-test-slave-69rg krb5kdc[22594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069555, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:55.508838 22843 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:25:55.509660 22843 webserver.cc:460] Webserver started at http://127.4.221.66:43785/ using document root <none> and password file <none>
I20260807 02:25:55.510263 22843 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:55.510354 22843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:55.510587 22843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:25:55.512584 22843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "1fd2370e8d97433fa7a2f92c6b4342ed"
format_stamp: "Formatted at 2026-08-07 02:25:55 on dist-test-slave-69rg"
server_key: "591c3202fdf4f5971577d7a0585ea727"
server_key_iv: "9fd4fa0e74b7dc6d9352a36dcfc6e2be"
server_key_version: "encryptionkey@0"
I20260807 02:25:55.513145 22843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "1fd2370e8d97433fa7a2f92c6b4342ed"
format_stamp: "Formatted at 2026-08-07 02:25:55 on dist-test-slave-69rg"
server_key: "591c3202fdf4f5971577d7a0585ea727"
server_key_iv: "9fd4fa0e74b7dc6d9352a36dcfc6e2be"
server_key_version: "encryptionkey@0"
I20260807 02:25:55.516863 22843 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260807 02:25:55.519342 22860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:55.520488 22843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 02:25:55.520639 22843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "1fd2370e8d97433fa7a2f92c6b4342ed"
format_stamp: "Formatted at 2026-08-07 02:25:55 on dist-test-slave-69rg"
server_key: "591c3202fdf4f5971577d7a0585ea727"
server_key_iv: "9fd4fa0e74b7dc6d9352a36dcfc6e2be"
server_key_version: "encryptionkey@0"
I20260807 02:25:55.520754 22843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:55.541262 22843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:55.544368 22843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:55.544579 22843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:55.545177 22843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:25:55.546082 22843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:25:55.546150 22843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:55.546231 22843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:25:55.546277 22843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:55.555388 22843 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:35803
I20260807 02:25:55.555419 22974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:35803 every 8 connection(s)
I20260807 02:25:55.556269 22843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:25:55.559024 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 22843
I20260807 02:25:55.559185 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:25:55.559535 4981 external_mini_cluster.cc:1428] Setting key 73361828d7dedfbd3f5dfd8a72748d0d
Aug 07 02:25:55 dist-test-slave-69rg krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069555, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
I20260807 02:25:55.570797 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:55.558176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:60049 (local address 127.4.221.126:40161)
0807 02:25:55.558344 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:55.558349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:55.559401 (+ 1052us) server_negotiation.cc:407] Connection header received
0807 02:25:55.560551 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:55.560554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:55.560597 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:55.560685 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:55.561915 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.562463 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.564046 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.564277 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.564366 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:55.567236 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:55.567240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:55.567242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:55.567266 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:55.568690 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:55.569228 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:55.569238 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:55.569240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:55.569314 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:55.569657 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:55.569661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:55.569663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:55.569926 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:55.570034 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:55.570507 (+ 473us) server_negotiation.cc:299] Negotiation successful
0807 02:25:55.570630 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 02:25:55.571588 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:55.558628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40161 (local address 127.4.221.66:60049)
0807 02:25:55.559229 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:55.559287 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:55.560362 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:55.560860 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:55.560870 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:55.561270 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:25:55.561721 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:55.561734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.562590 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.562593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:55.563909 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:55.563923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.564437 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.564445 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:55.564674 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:55.565281 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:25:55.565305 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:25:55.567059 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:25:55.568830 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:55.568836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:55.568850 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:55.569109 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:55.569415 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:55.569418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:55.569420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:55.569559 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:55.570032 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:25:55.570039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:25:55.570278 (+ 239us) client_negotiation.cc:769] Sending connection context
0807 02:25:55.570464 (+ 186us) client_negotiation.cc:240] Negotiation successful
0807 02:25:55.570823 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":171,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
I20260807 02:25:55.572726 22975 heartbeater.cc:347] Connected to a master server at 127.4.221.126:40161
I20260807 02:25:55.572973 22975 heartbeater.cc:464] Registering TS with master...
I20260807 02:25:55.573455 22975 heartbeater.cc:511] Master 127.4.221.126:40161 requested a full tablet report, sending...
I20260807 02:25:55.574602 22640 ts_manager.cc:194] Registered new tserver with Master: 1fd2370e8d97433fa7a2f92c6b4342ed (127.4.221.66:35803)
I20260807 02:25:55.575196 22640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:60049
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:25:55.589411 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40161
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:44461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:25:55.699935 22981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:55.700179 22981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:55.700224 22981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:55.704342 22981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:55.704409 22981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:55.704497 22981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:25:55.710008 22981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:55.711022 22981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:55.712066 22981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 02:25:55.722887 22981 server_base.cc:1034] running on GCE node
W20260807 02:25:55.722849 22990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:55.722852 22987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:55.722862 22988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:55.723498 22981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:55.724159 22981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:55.725355 22981 hybrid_clock.cc:648] HybridClock initialized: now 1786069555725336 us; error 32 us; skew 500 ppm
Aug 07 02:25:55 dist-test-slave-69rg krb5kdc[22594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069555, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:55.728091 22981 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:25:55.728924 22981 webserver.cc:460] Webserver started at http://127.4.221.67:33133/ using document root <none> and password file <none>
I20260807 02:25:55.729477 22981 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:55.729570 22981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:55.729800 22981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:25:55.732004 22981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "68d0a28b7bef49bb81743a918d63a760"
format_stamp: "Formatted at 2026-08-07 02:25:55 on dist-test-slave-69rg"
server_key: "8373b2212862061d57436f27603d6268"
server_key_iv: "efda5c7ded63c1999ed866e3716065bb"
server_key_version: "encryptionkey@0"
I20260807 02:25:55.732616 22981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "68d0a28b7bef49bb81743a918d63a760"
format_stamp: "Formatted at 2026-08-07 02:25:55 on dist-test-slave-69rg"
server_key: "8373b2212862061d57436f27603d6268"
server_key_iv: "efda5c7ded63c1999ed866e3716065bb"
server_key_version: "encryptionkey@0"
I20260807 02:25:55.736495 22981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 02:25:55.738925 22998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:55.740187 22981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 02:25:55.740327 22981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "68d0a28b7bef49bb81743a918d63a760"
format_stamp: "Formatted at 2026-08-07 02:25:55 on dist-test-slave-69rg"
server_key: "8373b2212862061d57436f27603d6268"
server_key_iv: "efda5c7ded63c1999ed866e3716065bb"
server_key_version: "encryptionkey@0"
I20260807 02:25:55.740443 22981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:55.766505 22981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:55.769373 22981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:55.769582 22981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:55.770233 22981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:25:55.771039 22981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:25:55.771113 22981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:55.771195 22981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:25:55.771232 22981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:55.780303 22981 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:41009
I20260807 02:25:55.780339 23117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:41009 every 8 connection(s)
I20260807 02:25:55.781167 22981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:25:55.786382 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 22981
I20260807 02:25:55.786516 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:25:55.786813 4981 external_mini_cluster.cc:1428] Setting key a959980b02482c377d69450d4a174842
Aug 07 02:25:55 dist-test-slave-69rg krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069555, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:25:55.793797 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:55.783001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:44033 (local address 127.4.221.126:40161)
0807 02:25:55.783149 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:55.783158 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:55.783940 (+ 782us) server_negotiation.cc:407] Connection header received
0807 02:25:55.784851 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:55.784854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:55.784917 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:55.785020 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:55.786220 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.786869 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.788135 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.788358 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.788467 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:55.790859 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:55.790862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:55.790864 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:55.790889 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:55.792088 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:55.792532 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:55.792535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:55.792536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:55.792606 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:55.792883 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:55.792885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:55.792886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:55.793047 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:55.793172 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:55.793581 (+ 409us) server_negotiation.cc:299] Negotiation successful
0807 02:25:55.793675 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 02:25:55.794663 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:55.783333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40161 (local address 127.4.221.67:44033)
0807 02:25:55.783787 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:55.783828 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:55.784622 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:55.785197 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:55.785204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:55.785543 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:25:55.786075 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:55.786089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.787009 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.787012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:55.787994 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:55.788004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.788516 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.788520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:55.788689 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:55.789206 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:25:55.789222 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:25:55.790712 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:25:55.792195 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:55.792199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:55.792210 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:55.792449 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:55.792682 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:55.792684 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:55.792686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:55.792807 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:55.793170 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:25:55.793174 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:25:55.793381 (+ 207us) client_negotiation.cc:769] Sending connection context
0807 02:25:55.793529 (+ 148us) client_negotiation.cc:240] Negotiation successful
0807 02:25:55.793782 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":111,"threads_started":1}
I20260807 02:25:55.795728 23119 heartbeater.cc:347] Connected to a master server at 127.4.221.126:40161
I20260807 02:25:55.795922 23119 heartbeater.cc:464] Registering TS with master...
I20260807 02:25:55.796370 23119 heartbeater.cc:511] Master 127.4.221.126:40161 requested a full tablet report, sending...
I20260807 02:25:55.797297 22640 ts_manager.cc:194] Registered new tserver with Master: 68d0a28b7bef49bb81743a918d63a760 (127.4.221.67:41009)
I20260807 02:25:55.797879 22640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:44033
I20260807 02:25:55.800455 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:25:55.810446 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:55.802579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.4.221.126:40161)
0807 02:25:55.802710 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:55.802715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:55.803116 (+ 401us) server_negotiation.cc:407] Connection header received
0807 02:25:55.803253 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:55.803256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:55.803307 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:55.803386 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:55.804124 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.804652 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.805428 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.805627 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.805701 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:55.807133 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:55.807138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:55.807141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:55.807166 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:55.808804 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:55.809232 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:55.809234 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:55.809235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:55.809306 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:55.809580 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:55.809582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:55.809583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:55.809787 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:55.809962 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:55.810163 (+ 201us) server_negotiation.cc:299] Negotiation successful
0807 02:25:55.810257 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 02:25:55.813257 22640 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:25:55.816397 22640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:25:55.831748 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:55.826022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.4.221.66:35803)
0807 02:25:55.826907 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:55.826916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:55.826935 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:25:55.826999 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:55.827003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:55.827144 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:55.827311 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:55.827565 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.828661 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.830314 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.831002 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.831110 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:55.831245 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:55.831477 (+ 232us) server_negotiation.cc:299] Negotiation successful
0807 02:25:55.831590 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":159,"threads_started":1}
I20260807 02:25:55.831809 23139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:55.825545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:35803 (local address 127.0.0.1:53778)
0807 02:25:55.826486 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:55.826499 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:55.826627 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:55.827291 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:55.827294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:55.827313 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:55.827428 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:55.827436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.828779 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.828782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:55.830190 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:55.830200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.831129 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.831133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:55.831267 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:55.831311 (+ 44us) client_negotiation.cc:769] Sending connection context
0807 02:25:55.831428 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 02:25:55.831541 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":117,"thread_start_us":122,"threads_started":1}
I20260807 02:25:55.832209 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:55.825236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:41009 (local address 127.0.0.1:59784)
0807 02:25:55.826465 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:55.826479 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:55.826653 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:55.827100 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:55.827105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:55.827127 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:55.827262 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:55.827269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.828734 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.828737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:55.830605 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:55.830618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.831798 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.831804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:55.831923 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:55.831951 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 02:25:55.832028 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 02:25:55.832086 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1173,"mutex_wait_us":94,"thread_start_us":91,"threads_started":1}
I20260807 02:25:55.832782 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:55.825247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.4.221.65:45337)
0807 02:25:55.827630 (+ 2383us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:55.827644 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:55.827662 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:25:55.827723 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:55.827726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:55.827856 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:55.828009 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:55.828275 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.829335 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.830993 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.832169 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.832301 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:55.832438 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:55.832525 (+ 87us) server_negotiation.cc:299] Negotiation successful
0807 02:25:55.832632 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2271,"thread_start_us":160,"threads_started":1}
I20260807 02:25:55.832789 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:55.825929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59784 (local address 127.4.221.67:41009)
0807 02:25:55.826282 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:55.826296 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:55.826598 (+ 302us) server_negotiation.cc:407] Connection header received
0807 02:25:55.826795 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:55.826800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:55.826951 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:55.827110 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:55.827384 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.828592 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.830749 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.831506 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.831605 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:55.831727 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:55.832516 (+ 789us) server_negotiation.cc:299] Negotiation successful
0807 02:25:55.832629 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":143,"threads_started":1}
I20260807 02:25:55.832732 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:55.825048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:45337 (local address 127.0.0.1:33128)
0807 02:25:55.825893 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:55.825950 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:55.826086 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:55.828005 (+ 1919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:55.828008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:55.828026 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:55.828144 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:55.828155 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.829508 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.829512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:55.830873 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:55.830881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:55.832301 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:55.832305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:55.832397 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:55.832420 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 02:25:55.832497 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 02:25:55.832569 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":43,"thread_start_us":106,"threads_started":1}
I20260807 02:25:55.835525 23047 tablet_service.cc:1467] Processing CreateTablet for tablet fb6683c8a8794f0591784fbc5ec42db5 (DEFAULT_TABLE table=test-table [id=7289ecc80d844917ac69910ff6d07700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:25:55.835794 22909 tablet_service.cc:1467] Processing CreateTablet for tablet fb6683c8a8794f0591784fbc5ec42db5 (DEFAULT_TABLE table=test-table [id=7289ecc80d844917ac69910ff6d07700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:25:55.835790 22770 tablet_service.cc:1467] Processing CreateTablet for tablet fb6683c8a8794f0591784fbc5ec42db5 (DEFAULT_TABLE table=test-table [id=7289ecc80d844917ac69910ff6d07700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:25:55.836558 23047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6683c8a8794f0591784fbc5ec42db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:25:55.836867 22909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6683c8a8794f0591784fbc5ec42db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:25:55.836867 22770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb6683c8a8794f0591784fbc5ec42db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:25:55.842561 23142 tablet_bootstrap.cc:492] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760: Bootstrap starting.
I20260807 02:25:55.844417 23144 tablet_bootstrap.cc:492] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592: Bootstrap starting.
I20260807 02:25:55.844769 23143 tablet_bootstrap.cc:492] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed: Bootstrap starting.
I20260807 02:25:55.844960 23142 tablet_bootstrap.cc:654] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760: Neither blocks nor log segments found. Creating new log.
I20260807 02:25:55.846310 23142 log.cc:824] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:55.847124 23144 tablet_bootstrap.cc:654] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592: Neither blocks nor log segments found. Creating new log.
I20260807 02:25:55.848116 23144 log.cc:824] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:55.848585 23143 tablet_bootstrap.cc:654] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed: Neither blocks nor log segments found. Creating new log.
I20260807 02:25:55.848960 23142 tablet_bootstrap.cc:492] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760: No bootstrap required, opened a new log
I20260807 02:25:55.849174 23142 ts_tablet_manager.cc:1397] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260807 02:25:55.849466 23143 log.cc:824] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:55.849934 23144 tablet_bootstrap.cc:492] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592: No bootstrap required, opened a new log
I20260807 02:25:55.850126 23144 ts_tablet_manager.cc:1397] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260807 02:25:55.851645 23143 tablet_bootstrap.cc:492] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed: No bootstrap required, opened a new log
I20260807 02:25:55.851895 23143 ts_tablet_manager.cc:1397] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260807 02:25:55.853256 23144 raft_consensus.cc:348] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd7e24ba08f744c8926d9727a9512592" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45337 } } peers { permanent_uuid: "68d0a28b7bef49bb81743a918d63a760" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41009 } } peers { permanent_uuid: "1fd2370e8d97433fa7a2f92c6b4342ed" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 35803 } }
I20260807 02:25:55.853489 23144 raft_consensus.cc:374] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:25:55.853570 23144 raft_consensus.cc:729] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd7e24ba08f744c8926d9727a9512592, State: Initialized, Role: FOLLOWER
I20260807 02:25:55.853657 23142 raft_consensus.cc:348] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd7e24ba08f744c8926d9727a9512592" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45337 } } peers { permanent_uuid: "68d0a28b7bef49bb81743a918d63a760" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41009 } } peers { permanent_uuid: "1fd2370e8d97433fa7a2f92c6b4342ed" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 35803 } }
I20260807 02:25:55.853986 23142 raft_consensus.cc:374] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:25:55.854054 23142 raft_consensus.cc:729] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68d0a28b7bef49bb81743a918d63a760, State: Initialized, Role: FOLLOWER
I20260807 02:25:55.854099 23144 consensus_queue.cc:260] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd7e24ba08f744c8926d9727a9512592" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45337 } } peers { permanent_uuid: "68d0a28b7bef49bb81743a918d63a760" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41009 } } peers { permanent_uuid: "1fd2370e8d97433fa7a2f92c6b4342ed" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 35803 } }
I20260807 02:25:55.854523 23142 consensus_queue.cc:260] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd7e24ba08f744c8926d9727a9512592" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45337 } } peers { permanent_uuid: "68d0a28b7bef49bb81743a918d63a760" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41009 } } peers { permanent_uuid: "1fd2370e8d97433fa7a2f92c6b4342ed" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 35803 } }
I20260807 02:25:55.854756 22838 heartbeater.cc:503] Master 127.4.221.126:40161 was elected leader, sending a full tablet report...
I20260807 02:25:55.855106 23144 ts_tablet_manager.cc:1428] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 02:25:55.855573 23142 ts_tablet_manager.cc:1428] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260807 02:25:55.855997 23119 heartbeater.cc:503] Master 127.4.221.126:40161 was elected leader, sending a full tablet report...
I20260807 02:25:55.856230 23143 raft_consensus.cc:348] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd7e24ba08f744c8926d9727a9512592" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45337 } } peers { permanent_uuid: "68d0a28b7bef49bb81743a918d63a760" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41009 } } peers { permanent_uuid: "1fd2370e8d97433fa7a2f92c6b4342ed" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 35803 } }
I20260807 02:25:55.856498 23143 raft_consensus.cc:374] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:25:55.856549 23143 raft_consensus.cc:729] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fd2370e8d97433fa7a2f92c6b4342ed, State: Initialized, Role: FOLLOWER
I20260807 02:25:55.856957 23143 consensus_queue.cc:260] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd7e24ba08f744c8926d9727a9512592" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45337 } } peers { permanent_uuid: "68d0a28b7bef49bb81743a918d63a760" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41009 } } peers { permanent_uuid: "1fd2370e8d97433fa7a2f92c6b4342ed" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 35803 } }
I20260807 02:25:55.858630 22975 heartbeater.cc:503] Master 127.4.221.126:40161 was elected leader, sending a full tablet report...
I20260807 02:25:55.858996 23143 ts_tablet_manager.cc:1428] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260807 02:25:56.032431 23121 tablet.cc:2369] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:25:56.049343 23151 raft_consensus.cc:482] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:25:56.049685 23151 raft_consensus.cc:504] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd7e24ba08f744c8926d9727a9512592" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45337 } } peers { permanent_uuid: "68d0a28b7bef49bb81743a918d63a760" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41009 } } peers { permanent_uuid: "1fd2370e8d97433fa7a2f92c6b4342ed" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 35803 } }
I20260807 02:25:56.051330 23151 leader_election.cc:288] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd7e24ba08f744c8926d9727a9512592 (127.4.221.65:45337), 68d0a28b7bef49bb81743a918d63a760 (127.4.221.67:41009)
I20260807 02:25:56.055560 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:56.051994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:56303 (local address 127.4.221.67:41009)
0807 02:25:56.052171 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:56.052174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:56.052191 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:25:56.052254 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:56.052256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:56.052301 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:56.052397 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:56.052714 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.053264 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.054460 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.054920 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.055061 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:56.055125 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:56.055318 (+ 193us) server_negotiation.cc:299] Negotiation successful
0807 02:25:56.055403 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260807 02:25:56.055559 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:56.051691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:41009 (local address 127.4.221.66:56303)
0807 02:25:56.051889 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:56.051912 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:56.052073 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:56.052435 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:56.052438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:56.052458 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:56.052597 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:56.052603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.053379 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.053393 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:56.054323 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:56.054333 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.055053 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.055056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:56.055177 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:56.055197 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 02:25:56.055295 (+ 98us) client_negotiation.cc:240] Negotiation successful
0807 02:25:56.055400 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260807 02:25:56.055996 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:56.051838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:46413 (local address 127.4.221.65:45337)
0807 02:25:56.052018 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:56.052024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:56.052369 (+ 345us) server_negotiation.cc:407] Connection header received
0807 02:25:56.052499 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:56.052502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:56.052577 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:56.052685 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:56.053022 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.053587 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.054823 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.055289 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.055434 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:56.055493 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:56.055768 (+ 275us) server_negotiation.cc:299] Negotiation successful
0807 02:25:56.055862 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 02:25:56.056421 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:56.051691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:45337 (local address 127.4.221.66:46413)
0807 02:25:56.052242 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:56.052254 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:56.052385 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:56.052694 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:56.052697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:56.052715 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:56.052868 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:56.052874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.053716 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.053722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:56.054681 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:56.054690 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.055434 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.055446 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:56.055568 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:56.055602 (+ 34us) client_negotiation.cc:769] Sending connection context
0807 02:25:56.056181 (+ 579us) client_negotiation.cc:240] Negotiation successful
0807 02:25:56.056280 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":156,"thread_start_us":121,"threads_started":1}
I20260807 02:25:56.056450 23067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6683c8a8794f0591784fbc5ec42db5" candidate_uuid: "1fd2370e8d97433fa7a2f92c6b4342ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68d0a28b7bef49bb81743a918d63a760" is_pre_election: true
I20260807 02:25:56.056860 23067 raft_consensus.cc:2454] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fd2370e8d97433fa7a2f92c6b4342ed in term 0.
I20260807 02:25:56.057215 22790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6683c8a8794f0591784fbc5ec42db5" candidate_uuid: "1fd2370e8d97433fa7a2f92c6b4342ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd7e24ba08f744c8926d9727a9512592" is_pre_election: true
W20260807 02:25:56.057512 22976 tablet.cc:2369] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:25:56.057614 22790 raft_consensus.cc:2454] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fd2370e8d97433fa7a2f92c6b4342ed in term 0.
I20260807 02:25:56.057551 22863 leader_election.cc:302] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd2370e8d97433fa7a2f92c6b4342ed, 68d0a28b7bef49bb81743a918d63a760; no voters:
I20260807 02:25:56.057940 23151 raft_consensus.cc:2781] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:25:56.058054 23151 raft_consensus.cc:482] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:25:56.058143 23151 raft_consensus.cc:3037] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:25:56.059974 23151 raft_consensus.cc:504] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd7e24ba08f744c8926d9727a9512592" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45337 } } peers { permanent_uuid: "68d0a28b7bef49bb81743a918d63a760" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41009 } } peers { permanent_uuid: "1fd2370e8d97433fa7a2f92c6b4342ed" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 35803 } }
I20260807 02:25:56.060515 23151 leader_election.cc:288] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [CANDIDATE]: Term 1 election: Requested vote from peers bd7e24ba08f744c8926d9727a9512592 (127.4.221.65:45337), 68d0a28b7bef49bb81743a918d63a760 (127.4.221.67:41009)
I20260807 02:25:56.061010 22790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6683c8a8794f0591784fbc5ec42db5" candidate_uuid: "1fd2370e8d97433fa7a2f92c6b4342ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd7e24ba08f744c8926d9727a9512592"
I20260807 02:25:56.061017 23067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb6683c8a8794f0591784fbc5ec42db5" candidate_uuid: "1fd2370e8d97433fa7a2f92c6b4342ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68d0a28b7bef49bb81743a918d63a760"
I20260807 02:25:56.061164 22790 raft_consensus.cc:3037] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:25:56.061164 23067 raft_consensus.cc:3037] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:25:56.062794 23067 raft_consensus.cc:2454] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fd2370e8d97433fa7a2f92c6b4342ed in term 1.
I20260807 02:25:56.062794 22790 raft_consensus.cc:2454] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fd2370e8d97433fa7a2f92c6b4342ed in term 1.
I20260807 02:25:56.063149 22863 leader_election.cc:302] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fd2370e8d97433fa7a2f92c6b4342ed, 68d0a28b7bef49bb81743a918d63a760; no voters:
I20260807 02:25:56.063372 23151 raft_consensus.cc:2781] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:25:56.063740 23151 raft_consensus.cc:686] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [term 1 LEADER]: Becoming Leader. State: Replica: 1fd2370e8d97433fa7a2f92c6b4342ed, State: Running, Role: LEADER
I20260807 02:25:56.064258 23151 consensus_queue.cc:237] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd7e24ba08f744c8926d9727a9512592" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45337 } } peers { permanent_uuid: "68d0a28b7bef49bb81743a918d63a760" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41009 } } peers { permanent_uuid: "1fd2370e8d97433fa7a2f92c6b4342ed" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 35803 } }
I20260807 02:25:56.068780 22639 catalog_manager.cc:5495] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fd2370e8d97433fa7a2f92c6b4342ed (127.4.221.66). New cstate: current_term: 1 leader_uuid: "1fd2370e8d97433fa7a2f92c6b4342ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd7e24ba08f744c8926d9727a9512592" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68d0a28b7bef49bb81743a918d63a760" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fd2370e8d97433fa7a2f92c6b4342ed" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 35803 } health_report { overall_health: HEALTHY } } }
W20260807 02:25:56.091706 22839 tablet.cc:2369] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:25:56.132529 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:56.128111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.4.221.66:35803)
0807 02:25:56.128285 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:56.128291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:56.128308 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:25:56.128551 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:56.128554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:56.128630 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:56.128756 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:25:56.129084 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.129660 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.130647 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.130888 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.131006 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:56.131367 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:25:56.131946 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:25:56.132096 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:56.132130 (+ 34us) server_negotiation.cc:299] Negotiation successful
0807 02:25:56.132203 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260807 02:25:56.139894 22790 raft_consensus.cc:1261] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592 [term 1 FOLLOWER]: Refusing update from remote peer 1fd2370e8d97433fa7a2f92c6b4342ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:25:56.139895 23067 raft_consensus.cc:1261] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760 [term 1 FOLLOWER]: Refusing update from remote peer 1fd2370e8d97433fa7a2f92c6b4342ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:25:56.140763 23153 consensus_queue.cc:1035] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "68d0a28b7bef49bb81743a918d63a760" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 41009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:25:56.141060 23151 consensus_queue.cc:1035] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd7e24ba08f744c8926d9727a9512592" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:25:56.151266 23157 mvcc.cc:205] Tried to move back new op lower bound from 7315740901939695616 to 7315740901643182080. Current Snapshot: MvccSnapshot[applied={T|T < 7315740901939695616}]
I20260807 02:25:56.161521 22640 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:25:56.161737 22640 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:25:56.164934 22640 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af9e094a3b0544dc96aa79b36a31bb88: Sending DeleteTablet for 3 replicas of tablet fb6683c8a8794f0591784fbc5ec42db5
I20260807 02:25:56.166075 23047 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6683c8a8794f0591784fbc5ec42db5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:25:56 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:59784
I20260807 02:25:56.166070 22770 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6683c8a8794f0591784fbc5ec42db5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:25:56 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:33128
I20260807 02:25:56.166788 23167 tablet_replica.cc:330] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592: stopping tablet replica
I20260807 02:25:56.167102 22909 tablet_service.cc:1514] Processing DeleteTablet for tablet fb6683c8a8794f0591784fbc5ec42db5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:25:56 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:53778
I20260807 02:25:56.167421 23167 raft_consensus.cc:2229] T fb6683c8a8794f0591784fbc5ec42db5 P bd7e24ba08f744c8926d9727a9512592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:25:56.167804 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 22704
I20260807 02:25:56.170174 23166 tablet_replica.cc:330] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760: stopping tablet replica
I20260807 02:25:56.170607 23166 raft_consensus.cc:2229] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:25:56.171005 23166 raft_consensus.cc:2258] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:25:56.173132 23166 ts_tablet_manager.cc:1905] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:25:56.177680 23166 ts_tablet_manager.cc:1918] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:25:56.177848 23166 log.cc:1196] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/fb6683c8a8794f0591784fbc5ec42db5
I20260807 02:25:56.178309 23168 tablet_replica.cc:330] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed: stopping tablet replica
I20260807 02:25:56.178342 23166 ts_tablet_manager.cc:1939] T fb6683c8a8794f0591784fbc5ec42db5 P 68d0a28b7bef49bb81743a918d63a760: Deleting consensus metadata
I20260807 02:25:56.178844 23168 raft_consensus.cc:2229] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:25:56.179458 23168 raft_consensus.cc:2258] T fb6683c8a8794f0591784fbc5ec42db5 P 1fd2370e8d97433fa7a2f92c6b4342ed [term 1 FOLLOWER]: Raft consensus is shut down!
W20260807 02:25:56.179658 22627 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:45337 (error 108)
I20260807 02:25:56.180364 22626 catalog_manager.cc:4841] TS 68d0a28b7bef49bb81743a918d63a760 (127.4.221.67:41009): tablet fb6683c8a8794f0591784fbc5ec42db5 (table test-table [id=7289ecc80d844917ac69910ff6d07700]) successfully deleted
I20260807 02:25:56.180847 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 22843
I20260807 02:25:56.184062 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:56.183708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:45337 (local address 127.0.0.1:33138)
0807 02:25:56.183881 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:56.183963 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260807 02:25:56.184376 22627 catalog_manager.cc:4584] TS bd7e24ba08f744c8926d9727a9512592 (127.4.221.65:45337): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fb6683c8a8794f0591784fbc5ec42db5: Network error: Client connection negotiation failed: client connection to 127.4.221.65:45337: connect: Connection refused (error 111)
I20260807 02:25:56.189231 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 22981
I20260807 02:25:56.189648 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:56.189245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:35803 (local address 127.0.0.1:53800)
0807 02:25:56.189448 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:56.189523 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.66:35803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260807 02:25:56.197078 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 22607
2026-08-07T02:25:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2560 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-07T02:25:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:25:56Z Disabled control of system clock
I20260807 02:25:56.235163 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:38855
--webserver_interface=127.4.221.126
--webserver_port=0
--builtin_ntp_servers=127.4.221.84:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:38855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 02:25:56.362093 23173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:56.362414 23173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:56.362453 23173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:56.366551 23173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:25:56.366614 23173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:56.366631 23173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:56.366647 23173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:25:56.366663 23173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:25:56.371482 23173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:38855
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:38855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:56.372816 23173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:56.374512 23173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:56.384961 23183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:56.384948 23180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:56.384922 23181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:56.385277 23173 server_base.cc:1034] running on GCE node
I20260807 02:25:56.386191 23173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:56.387719 23173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:56.388947 23173 hybrid_clock.cc:648] HybridClock initialized: now 1786069556388918 us; error 45 us; skew 500 ppm
I20260807 02:25:56.391798 23173 webserver.cc:460] Webserver started at http://127.4.221.126:37265/ using document root <none> and password file <none>
I20260807 02:25:56.392580 23173 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:56.392694 23173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:56.393006 23173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:25:56.395746 23173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "0346130b74e94684a039c9eabf6c5a7d"
format_stamp: "Formatted at 2026-08-07 02:25:56 on dist-test-slave-69rg"
server_key: "80c4c4834abcab858b51404a93a6e3bd"
server_key_iv: "a87d79cad49bbcb6a32471e9088fef5f"
server_key_version: "encryptionkey@0"
I20260807 02:25:56.396478 23173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "0346130b74e94684a039c9eabf6c5a7d"
format_stamp: "Formatted at 2026-08-07 02:25:56 on dist-test-slave-69rg"
server_key: "80c4c4834abcab858b51404a93a6e3bd"
server_key_iv: "a87d79cad49bbcb6a32471e9088fef5f"
server_key_version: "encryptionkey@0"
I20260807 02:25:56.401384 23173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260807 02:25:56.404631 23191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:56.406141 23173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:25:56.406339 23173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "0346130b74e94684a039c9eabf6c5a7d"
format_stamp: "Formatted at 2026-08-07 02:25:56 on dist-test-slave-69rg"
server_key: "80c4c4834abcab858b51404a93a6e3bd"
server_key_iv: "a87d79cad49bbcb6a32471e9088fef5f"
server_key_version: "encryptionkey@0"
I20260807 02:25:56.406536 23173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:56.428956 23173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:56.430037 23173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:56.430325 23173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:56.440058 23173 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:38855
I20260807 02:25:56.440065 23245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:38855 every 8 connection(s)
I20260807 02:25:56.441396 23173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:25:56.442824 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 23173
I20260807 02:25:56.442998 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:25:56.443387 4981 external_mini_cluster.cc:1428] Setting key aaeeeea9609681afa17b6a60b98cc997
I20260807 02:25:56.446074 23246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:25:56.453722 23246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d: Bootstrap starting.
I20260807 02:25:56.454219 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:56.445118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.4.221.126:38855)
0807 02:25:56.445855 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:56.445873 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:56.445916 (+ 43us) server_negotiation.cc:407] Connection header received
0807 02:25:56.446715 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:56.446734 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:56.447164 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:56.447485 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:56.447863 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.448939 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.450328 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.450873 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.451004 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:56.452031 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:56.452036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:56.452050 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:25:56.452524 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:56.452587 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:25:56.452608 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:56.452688 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:56.452862 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 02:25:56.453192 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":165,"threads_started":1}
I20260807 02:25:56.456625 23246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d: Neither blocks nor log segments found. Creating new log.
I20260807 02:25:56.457549 23246 log.cc:824] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:56.460036 23246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d: No bootstrap required, opened a new log
I20260807 02:25:56.464210 23246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0346130b74e94684a039c9eabf6c5a7d" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38855 } }
I20260807 02:25:56.464542 23246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:25:56.464654 23246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0346130b74e94684a039c9eabf6c5a7d, State: Initialized, Role: FOLLOWER
I20260807 02:25:56.465175 23246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0346130b74e94684a039c9eabf6c5a7d" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38855 } }
I20260807 02:25:56.465348 23246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:25:56.465447 23246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:25:56.465579 23246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:25:56.466851 23246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0346130b74e94684a039c9eabf6c5a7d" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38855 } }
I20260807 02:25:56.467255 23246 leader_election.cc:302] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0346130b74e94684a039c9eabf6c5a7d; no voters:
I20260807 02:25:56.467711 23246 leader_election.cc:288] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:25:56.467877 23251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:25:56.468158 23251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [term 1 LEADER]: Becoming Leader. State: Replica: 0346130b74e94684a039c9eabf6c5a7d, State: Running, Role: LEADER
I20260807 02:25:56.468511 23251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0346130b74e94684a039c9eabf6c5a7d" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38855 } }
I20260807 02:25:56.469309 23246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:25:56.470427 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0346130b74e94684a039c9eabf6c5a7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0346130b74e94684a039c9eabf6c5a7d" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38855 } } }
I20260807 02:25:56.470574 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [sys.catalog]: This master's current role is: LEADER
I20260807 02:25:56.470885 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0346130b74e94684a039c9eabf6c5a7d. Latest consensus state: current_term: 1 leader_uuid: "0346130b74e94684a039c9eabf6c5a7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0346130b74e94684a039c9eabf6c5a7d" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 38855 } } }
I20260807 02:25:56.470984 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d [sys.catalog]: This master's current role is: LEADER
I20260807 02:25:56.471179 23260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:25:56.474912 23260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:25:56.482522 23260 catalog_manager.cc:1347] Generated new cluster ID: cef7fa34143e44a2919fe14021de66f2
I20260807 02:25:56.482615 23260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:25:56.494935 23260 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:25:56.496474 23260 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:25:56.510215 23260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d: Generated new TSK 0
I20260807 02:25:56.511443 23260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:25:56.521603 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:38855
--builtin_ntp_servers=127.4.221.84:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 02:25:56.654322 23270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:56.654659 23270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:56.654706 23270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:56.658785 23270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:56.658840 23270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:56.658927 23270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:25:56.663708 23270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:56.665071 23270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:56.666832 23270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:56.678490 23276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:56.678503 23277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:56.678486 23279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:56.678692 23270 server_base.cc:1034] running on GCE node
I20260807 02:25:56.679301 23270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:56.680083 23270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:56.681246 23270 hybrid_clock.cc:648] HybridClock initialized: now 1786069556681213 us; error 47 us; skew 500 ppm
I20260807 02:25:56.683454 23270 webserver.cc:460] Webserver started at http://127.4.221.65:45645/ using document root <none> and password file <none>
I20260807 02:25:56.684085 23270 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:56.684149 23270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:56.684341 23270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:25:56.686796 23270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "252e3b9ad7c5473484490867689acc9c"
format_stamp: "Formatted at 2026-08-07 02:25:56 on dist-test-slave-69rg"
server_key: "168486ef1ec9533ed24b315989e52053"
server_key_iv: "8974b7a95535e91bcbd83a5a2af82c10"
server_key_version: "encryptionkey@0"
I20260807 02:25:56.687384 23270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "252e3b9ad7c5473484490867689acc9c"
format_stamp: "Formatted at 2026-08-07 02:25:56 on dist-test-slave-69rg"
server_key: "168486ef1ec9533ed24b315989e52053"
server_key_iv: "8974b7a95535e91bcbd83a5a2af82c10"
server_key_version: "encryptionkey@0"
I20260807 02:25:56.691672 23270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:25:56.694528 23285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:56.695991 23270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:25:56.696102 23270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "252e3b9ad7c5473484490867689acc9c"
format_stamp: "Formatted at 2026-08-07 02:25:56 on dist-test-slave-69rg"
server_key: "168486ef1ec9533ed24b315989e52053"
server_key_iv: "8974b7a95535e91bcbd83a5a2af82c10"
server_key_version: "encryptionkey@0"
I20260807 02:25:56.696206 23270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:56.713137 23270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:56.713966 23270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:56.714143 23270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:56.714773 23270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:25:56.715829 23270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:25:56.715881 23270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:56.715927 23270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:25:56.715950 23270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:56.727620 23270 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:38783
I20260807 02:25:56.727653 23405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:38783 every 8 connection(s)
I20260807 02:25:56.728718 23270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:25:56.731079 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 23270
I20260807 02:25:56.731215 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:25:56.731570 4981 external_mini_cluster.cc:1428] Setting key 3caeacc534e37914f8611b73a3cf0a79
I20260807 02:25:56.734252 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:38855
--builtin_ntp_servers=127.4.221.84:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 02:25:56.741497 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:56.731101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:53707 (local address 127.4.221.126:38855)
0807 02:25:56.731265 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:56.731269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:56.732849 (+ 1580us) server_negotiation.cc:407] Connection header received
0807 02:25:56.734900 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:56.734907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:56.734981 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:56.735083 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:56.736128 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.736906 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.738835 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.739095 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.739198 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:56.740862 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:56.740865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:56.740868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:25:56.740884 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:56.740909 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:25:56.740921 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:56.740978 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:56.741216 (+ 238us) server_negotiation.cc:299] Negotiation successful
0807 02:25:56.741314 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:25:56.743180 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:56.731707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:38855 (local address 127.4.221.65:53707)
0807 02:25:56.732670 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:56.732733 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:56.734665 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:56.735269 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:56.735276 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:56.735369 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:25:56.735977 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:56.735991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.737051 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.737056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:56.738699 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:56.738715 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.739284 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.739291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:56.739558 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:56.740452 (+ 894us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:25:56.740457 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:25:56.740532 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:25:56.740656 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:25:56.741010 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:25:56.741024 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:25:56.741034 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 02:25:56.741208 (+ 174us) client_negotiation.cc:240] Negotiation successful
0807 02:25:56.741615 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":185,"threads_started":1}
I20260807 02:25:56.744791 23406 heartbeater.cc:347] Connected to a master server at 127.4.221.126:38855
I20260807 02:25:56.745091 23406 heartbeater.cc:464] Registering TS with master...
I20260807 02:25:56.745744 23406 heartbeater.cc:511] Master 127.4.221.126:38855 requested a full tablet report, sending...
I20260807 02:25:56.747630 23208 ts_manager.cc:194] Registered new tserver with Master: 252e3b9ad7c5473484490867689acc9c (127.4.221.65:38783)
I20260807 02:25:56.748957 23208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.221.65:53707
W20260807 02:25:56.870183 23409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:56.870596 23409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:56.870679 23409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:56.875173 23409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:56.875263 23409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:56.875365 23409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:25:56.880441 23409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:56.881917 23409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:56.883697 23409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:56.894984 23415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:56.894980 23418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:56.894980 23416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:56.895401 23409 server_base.cc:1034] running on GCE node
I20260807 02:25:56.895962 23409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:56.896770 23409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:56.897981 23409 hybrid_clock.cc:648] HybridClock initialized: now 1786069556897964 us; error 28 us; skew 500 ppm
I20260807 02:25:56.900471 23409 webserver.cc:460] Webserver started at http://127.4.221.66:39015/ using document root <none> and password file <none>
I20260807 02:25:56.901252 23409 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:56.901361 23409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:56.901659 23409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:25:56.904479 23409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "51ed2b7b3ae84f809b06ffe0e909904b"
format_stamp: "Formatted at 2026-08-07 02:25:56 on dist-test-slave-69rg"
server_key: "574e80c9acdef9f743eb911d95b7aec0"
server_key_iv: "63c2ebc553758b17ea976f7cfd60f51b"
server_key_version: "encryptionkey@0"
I20260807 02:25:56.905176 23409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "51ed2b7b3ae84f809b06ffe0e909904b"
format_stamp: "Formatted at 2026-08-07 02:25:56 on dist-test-slave-69rg"
server_key: "574e80c9acdef9f743eb911d95b7aec0"
server_key_iv: "63c2ebc553758b17ea976f7cfd60f51b"
server_key_version: "encryptionkey@0"
I20260807 02:25:56.909881 23409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260807 02:25:56.913123 23424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:56.914820 23409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260807 02:25:56.914987 23409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "51ed2b7b3ae84f809b06ffe0e909904b"
format_stamp: "Formatted at 2026-08-07 02:25:56 on dist-test-slave-69rg"
server_key: "574e80c9acdef9f743eb911d95b7aec0"
server_key_iv: "63c2ebc553758b17ea976f7cfd60f51b"
server_key_version: "encryptionkey@0"
I20260807 02:25:56.915119 23409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:56.942293 23409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:56.943225 23409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:56.943485 23409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:56.944238 23409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:25:56.945477 23409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:25:56.945552 23409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:56.945634 23409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:25:56.945672 23409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:56.955989 23409 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:40533
I20260807 02:25:56.956015 23545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:40533 every 8 connection(s)
I20260807 02:25:56.957070 23409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:25:56.964481 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 23409
I20260807 02:25:56.964615 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:25:56.964977 4981 external_mini_cluster.cc:1428] Setting key 7d64aae386f4d3dd69c1bb37bf9d84ea
I20260807 02:25:56.967387 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:38855
--builtin_ntp_servers=127.4.221.84:42409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 02:25:56.968441 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:56.959299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:46921 (local address 127.4.221.126:38855)
0807 02:25:56.959467 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:56.959472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:56.960743 (+ 1271us) server_negotiation.cc:407] Connection header received
0807 02:25:56.962367 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:56.962370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:56.962430 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:56.962520 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:56.963516 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.964079 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.965885 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.966168 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.966314 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:56.967858 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:56.967862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:56.967872 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:25:56.967885 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:56.967907 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:25:56.967915 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:56.968005 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:56.968192 (+ 187us) server_negotiation.cc:299] Negotiation successful
0807 02:25:56.968276 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260807 02:25:56.969630 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:56.959784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:38855 (local address 127.4.221.66:46921)
0807 02:25:56.960561 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:56.960607 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:56.962145 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:56.962700 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:56.962707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:56.962797 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:25:56.963377 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:56.963390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.964215 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.964220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:56.965602 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:56.965613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:56.966362 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:56.966368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:56.966703 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:56.967517 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:25:56.967520 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:25:56.967595 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:25:56.967689 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:25:56.968019 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:25:56.968031 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:25:56.968037 (+ 6us) client_negotiation.cc:769] Sending connection context
0807 02:25:56.968192 (+ 155us) client_negotiation.cc:240] Negotiation successful
0807 02:25:56.968527 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":146,"threads_started":1}
I20260807 02:25:56.971242 23546 heartbeater.cc:347] Connected to a master server at 127.4.221.126:38855
I20260807 02:25:56.971526 23546 heartbeater.cc:464] Registering TS with master...
I20260807 02:25:56.972246 23546 heartbeater.cc:511] Master 127.4.221.126:38855 requested a full tablet report, sending...
I20260807 02:25:56.973542 23208 ts_manager.cc:194] Registered new tserver with Master: 51ed2b7b3ae84f809b06ffe0e909904b (127.4.221.66:40533)
I20260807 02:25:56.974593 23208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.221.66:46921
W20260807 02:25:57.098436 23549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:57.098682 23549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:57.098721 23549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:57.102905 23549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:57.102957 23549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:57.103009 23549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:25:57.107666 23549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:42409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:38855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:57.108683 23549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:57.110072 23549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:57.119876 23555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:57.119966 23559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:57.120024 23549 server_base.cc:1034] running on GCE node
W20260807 02:25:57.119947 23556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:57.120563 23549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:57.121210 23549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:57.122396 23549 hybrid_clock.cc:648] HybridClock initialized: now 1786069557122381 us; error 32 us; skew 500 ppm
I20260807 02:25:57.124264 23549 webserver.cc:460] Webserver started at http://127.4.221.67:43169/ using document root <none> and password file <none>
I20260807 02:25:57.124851 23549 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:57.124944 23549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:57.125191 23549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:25:57.127290 23549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "08ad44a954ee4ce6a5c514450ada6919"
format_stamp: "Formatted at 2026-08-07 02:25:57 on dist-test-slave-69rg"
server_key: "ced54bad04ba1aefb850836c54fddf97"
server_key_iv: "0c84b30c4c46a7fe8700fc65591a5f13"
server_key_version: "encryptionkey@0"
I20260807 02:25:57.127867 23549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "08ad44a954ee4ce6a5c514450ada6919"
format_stamp: "Formatted at 2026-08-07 02:25:57 on dist-test-slave-69rg"
server_key: "ced54bad04ba1aefb850836c54fddf97"
server_key_iv: "0c84b30c4c46a7fe8700fc65591a5f13"
server_key_version: "encryptionkey@0"
I20260807 02:25:57.131563 23549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 02:25:57.134037 23565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:57.135365 23549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:25:57.135505 23549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "08ad44a954ee4ce6a5c514450ada6919"
format_stamp: "Formatted at 2026-08-07 02:25:57 on dist-test-slave-69rg"
server_key: "ced54bad04ba1aefb850836c54fddf97"
server_key_iv: "0c84b30c4c46a7fe8700fc65591a5f13"
server_key_version: "encryptionkey@0"
I20260807 02:25:57.135604 23549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:57.155460 23549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:57.156103 23549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:57.156302 23549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:57.156880 23549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:25:57.157877 23549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:25:57.157929 23549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:57.157969 23549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:25:57.158017 23549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:57.166600 23549 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:37403
I20260807 02:25:57.166695 23685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:37403 every 8 connection(s)
I20260807 02:25:57.167603 23549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:25:57.176561 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 23549
I20260807 02:25:57.176695 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:25:57.177007 4981 external_mini_cluster.cc:1428] Setting key e4ff61872e9030c5927aa9467ed7f5bd
I20260807 02:25:57.177160 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:57.169534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:55533 (local address 127.4.221.126:38855)
0807 02:25:57.169693 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:57.169698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:57.170843 (+ 1145us) server_negotiation.cc:407] Connection header received
0807 02:25:57.172360 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:57.172363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:57.172427 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:57.172509 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:57.173440 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.174025 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.175245 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.175456 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.175537 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:57.176580 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:57.176584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:57.176586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:25:57.176599 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:57.176655 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:25:57.176670 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:57.176745 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:57.176931 (+ 186us) server_negotiation.cc:299] Negotiation successful
0807 02:25:57.176995 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 02:25:57.178283 23688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:57.169970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:38855 (local address 127.4.221.67:55533)
0807 02:25:57.170717 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:57.170757 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:57.172103 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:57.172706 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:57.172713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:57.172800 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:25:57.173302 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:57.173314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.174202 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.174205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:57.175081 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:57.175090 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.175603 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.175607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:57.175752 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:57.176303 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:25:57.176305 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:25:57.176342 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:25:57.176401 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:25:57.176789 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:25:57.176798 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:25:57.176802 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 02:25:57.176929 (+ 127us) client_negotiation.cc:240] Negotiation successful
0807 02:25:57.177243 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":143,"threads_started":1}
I20260807 02:25:57.179375 23686 heartbeater.cc:347] Connected to a master server at 127.4.221.126:38855
I20260807 02:25:57.179576 23686 heartbeater.cc:464] Registering TS with master...
I20260807 02:25:57.180025 23686 heartbeater.cc:511] Master 127.4.221.126:38855 requested a full tablet report, sending...
I20260807 02:25:57.180964 23208 ts_manager.cc:194] Registered new tserver with Master: 08ad44a954ee4ce6a5c514450ada6919 (127.4.221.67:37403)
I20260807 02:25:57.181531 23208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.221.67:55533
I20260807 02:25:57.192124 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:25:57.198211 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:57.194368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44106 (local address 127.4.221.126:38855)
0807 02:25:57.194459 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:57.194462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:57.194711 (+ 249us) server_negotiation.cc:407] Connection header received
0807 02:25:57.194868 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:57.194871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:57.194921 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:57.195001 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:57.195293 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.195793 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.196659 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.196884 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.196994 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:57.197778 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:57.197782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:57.197785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:25:57.197797 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:57.197842 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:25:57.197854 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:57.197920 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:57.198007 (+ 87us) server_negotiation.cc:299] Negotiation successful
0807 02:25:57.198055 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260807 02:25:57.200886 23208 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:25:57.204213 23208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:25:57.219208 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:57.213945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38386 (local address 127.4.221.66:40533)
0807 02:25:57.214587 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:57.214606 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:57.214644 (+ 38us) server_negotiation.cc:407] Connection header received
0807 02:25:57.214734 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:57.214741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:57.214927 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:57.215088 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:57.215278 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.216060 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.217637 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.218487 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.218601 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:57.218806 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:57.218912 (+ 106us) server_negotiation.cc:299] Negotiation successful
0807 02:25:57.219015 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":121,"threads_started":1}
I20260807 02:25:57.219203 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:57.213390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:40533 (local address 127.0.0.1:38386)
0807 02:25:57.214212 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:57.214226 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:57.214307 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:57.215074 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:57.215077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:57.215099 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:57.215183 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:57.215189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.216179 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.216184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:57.217507 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:57.217517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.218611 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.218615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:57.218728 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:57.218764 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 02:25:57.218881 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 02:25:57.219050 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":44,"thread_start_us":92,"threads_started":1}
I20260807 02:25:57.219405 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:57.213164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:38783 (local address 127.0.0.1:55050)
0807 02:25:57.213736 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:57.213753 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:57.213867 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:57.214790 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:57.214796 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:57.214819 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:57.214999 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:57.215010 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.216642 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.216645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:57.217730 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:57.217737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.218861 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.218865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:57.219024 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:57.219091 (+ 67us) client_negotiation.cc:769] Sending connection context
0807 02:25:57.219171 (+ 80us) client_negotiation.cc:240] Negotiation successful
0807 02:25:57.219240 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":85,"thread_start_us":92,"threads_started":1}
I20260807 02:25:57.219976 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:57.212962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:37403 (local address 127.0.0.1:56264)
0807 02:25:57.213311 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:57.213362 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:57.213481 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:57.214851 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:57.214854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:57.214874 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:57.214985 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:57.214994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.216661 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.216667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:57.217791 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:57.217801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.218861 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.218865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:57.219013 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:57.219720 (+ 707us) client_negotiation.cc:769] Sending connection context
0807 02:25:57.219787 (+ 67us) client_negotiation.cc:240] Negotiation successful
0807 02:25:57.219857 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":645,"thread_start_us":82,"threads_started":1}
I20260807 02:25:57.220078 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:57.213643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.4.221.67:37403)
0807 02:25:57.214441 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:57.214451 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:57.214470 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:25:57.214536 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:57.214540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:57.214705 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:57.215176 (+ 471us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:57.215330 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.216523 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.217950 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.218717 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.219129 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:57.219312 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:57.219822 (+ 510us) server_negotiation.cc:299] Negotiation successful
0807 02:25:57.219920 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":137,"threads_started":1}
I20260807 02:25:57.220633 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:57.213755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55050 (local address 127.4.221.65:38783)
0807 02:25:57.214210 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:57.214217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:57.214243 (+ 26us) server_negotiation.cc:407] Connection header received
0807 02:25:57.214334 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:57.214340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:57.214554 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:57.214788 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:57.215325 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.216489 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.217859 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.218712 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.220126 (+ 1414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:57.220323 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:57.220376 (+ 53us) server_negotiation.cc:299] Negotiation successful
0807 02:25:57.220446 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":146,"threads_started":1}
I20260807 02:25:57.222983 23617 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6e6ced51e3445aab39ead06f01049d (DEFAULT_TABLE table=test-table [id=8881a17df0ce4666bbd6a4453e4e8333]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:25:57.222981 23477 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6e6ced51e3445aab39ead06f01049d (DEFAULT_TABLE table=test-table [id=8881a17df0ce4666bbd6a4453e4e8333]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:25:57.223182 23337 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6e6ced51e3445aab39ead06f01049d (DEFAULT_TABLE table=test-table [id=8881a17df0ce4666bbd6a4453e4e8333]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:25:57.224131 23477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6e6ced51e3445aab39ead06f01049d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:25:57.224135 23617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6e6ced51e3445aab39ead06f01049d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:25:57.224175 23337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6e6ced51e3445aab39ead06f01049d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:25:57.230938 23704 tablet_bootstrap.cc:492] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c: Bootstrap starting.
I20260807 02:25:57.232242 23705 tablet_bootstrap.cc:492] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b: Bootstrap starting.
I20260807 02:25:57.233965 23704 tablet_bootstrap.cc:654] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c: Neither blocks nor log segments found. Creating new log.
I20260807 02:25:57.234169 23706 tablet_bootstrap.cc:492] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919: Bootstrap starting.
I20260807 02:25:57.234314 23705 tablet_bootstrap.cc:654] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b: Neither blocks nor log segments found. Creating new log.
I20260807 02:25:57.234920 23704 log.cc:824] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:57.235064 23705 log.cc:824] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:57.236307 23706 tablet_bootstrap.cc:654] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919: Neither blocks nor log segments found. Creating new log.
I20260807 02:25:57.236902 23704 tablet_bootstrap.cc:492] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c: No bootstrap required, opened a new log
I20260807 02:25:57.237187 23704 ts_tablet_manager.cc:1397] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260807 02:25:57.237234 23706 log.cc:824] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:57.237399 23705 tablet_bootstrap.cc:492] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b: No bootstrap required, opened a new log
I20260807 02:25:57.237753 23705 ts_tablet_manager.cc:1397] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260807 02:25:57.239236 23706 tablet_bootstrap.cc:492] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919: No bootstrap required, opened a new log
I20260807 02:25:57.239403 23706 ts_tablet_manager.cc:1397] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260807 02:25:57.241405 23705 raft_consensus.cc:348] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ad44a954ee4ce6a5c514450ada6919" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37403 } } peers { permanent_uuid: "252e3b9ad7c5473484490867689acc9c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38783 } } peers { permanent_uuid: "51ed2b7b3ae84f809b06ffe0e909904b" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40533 } }
I20260807 02:25:57.241649 23705 raft_consensus.cc:374] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:25:57.241711 23705 raft_consensus.cc:729] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51ed2b7b3ae84f809b06ffe0e909904b, State: Initialized, Role: FOLLOWER
I20260807 02:25:57.241760 23704 raft_consensus.cc:348] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ad44a954ee4ce6a5c514450ada6919" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37403 } } peers { permanent_uuid: "252e3b9ad7c5473484490867689acc9c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38783 } } peers { permanent_uuid: "51ed2b7b3ae84f809b06ffe0e909904b" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40533 } }
I20260807 02:25:57.242024 23704 raft_consensus.cc:374] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:25:57.242094 23704 raft_consensus.cc:729] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252e3b9ad7c5473484490867689acc9c, State: Initialized, Role: FOLLOWER
I20260807 02:25:57.242240 23705 consensus_queue.cc:260] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ad44a954ee4ce6a5c514450ada6919" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37403 } } peers { permanent_uuid: "252e3b9ad7c5473484490867689acc9c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38783 } } peers { permanent_uuid: "51ed2b7b3ae84f809b06ffe0e909904b" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40533 } }
I20260807 02:25:57.242496 23704 consensus_queue.cc:260] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ad44a954ee4ce6a5c514450ada6919" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37403 } } peers { permanent_uuid: "252e3b9ad7c5473484490867689acc9c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38783 } } peers { permanent_uuid: "51ed2b7b3ae84f809b06ffe0e909904b" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40533 } }
I20260807 02:25:57.242985 23546 heartbeater.cc:503] Master 127.4.221.126:38855 was elected leader, sending a full tablet report...
I20260807 02:25:57.243292 23705 ts_tablet_manager.cc:1428] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 02:25:57.243415 23704 ts_tablet_manager.cc:1428] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260807 02:25:57.243515 23706 raft_consensus.cc:348] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ad44a954ee4ce6a5c514450ada6919" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37403 } } peers { permanent_uuid: "252e3b9ad7c5473484490867689acc9c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38783 } } peers { permanent_uuid: "51ed2b7b3ae84f809b06ffe0e909904b" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40533 } }
I20260807 02:25:57.243746 23706 raft_consensus.cc:374] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:25:57.243755 23406 heartbeater.cc:503] Master 127.4.221.126:38855 was elected leader, sending a full tablet report...
I20260807 02:25:57.243799 23706 raft_consensus.cc:729] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ad44a954ee4ce6a5c514450ada6919, State: Initialized, Role: FOLLOWER
I20260807 02:25:57.244201 23706 consensus_queue.cc:260] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ad44a954ee4ce6a5c514450ada6919" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37403 } } peers { permanent_uuid: "252e3b9ad7c5473484490867689acc9c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38783 } } peers { permanent_uuid: "51ed2b7b3ae84f809b06ffe0e909904b" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40533 } }
I20260807 02:25:57.244817 23686 heartbeater.cc:503] Master 127.4.221.126:38855 was elected leader, sending a full tablet report...
I20260807 02:25:57.245461 23706 ts_tablet_manager.cc:1428] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 02:25:57.400441 23711 raft_consensus.cc:482] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:25:57.400723 23711 raft_consensus.cc:504] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ad44a954ee4ce6a5c514450ada6919" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37403 } } peers { permanent_uuid: "252e3b9ad7c5473484490867689acc9c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38783 } } peers { permanent_uuid: "51ed2b7b3ae84f809b06ffe0e909904b" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40533 } }
I20260807 02:25:57.402231 23711 leader_election.cc:288] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08ad44a954ee4ce6a5c514450ada6919 (127.4.221.67:37403), 51ed2b7b3ae84f809b06ffe0e909904b (127.4.221.66:40533)
I20260807 02:25:57.407632 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:57.402641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:52961 (local address 127.4.221.67:37403)
0807 02:25:57.402783 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:57.402790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:57.402871 (+ 81us) server_negotiation.cc:407] Connection header received
0807 02:25:57.403164 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:57.403169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:57.403228 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:57.403308 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:57.403679 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.404506 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.406328 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.406942 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.407069 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:57.407123 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:57.407419 (+ 296us) server_negotiation.cc:299] Negotiation successful
0807 02:25:57.407496 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:25:57.407632 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:57.402863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:37775 (local address 127.4.221.66:40533)
0807 02:25:57.403004 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:57.403010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:57.403258 (+ 248us) server_negotiation.cc:407] Connection header received
0807 02:25:57.403425 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:57.403430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:57.403494 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:57.403558 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:57.404065 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.404760 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.406358 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.406950 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.407081 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:57.407123 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:57.407420 (+ 297us) server_negotiation.cc:299] Negotiation successful
0807 02:25:57.407496 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 02:25:57.407693 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:57.402594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:40533 (local address 127.4.221.65:37775)
0807 02:25:57.403138 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:57.403156 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:57.403279 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:57.403751 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:57.403754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:57.403777 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:57.403918 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:57.403925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.404874 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.404878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:57.406218 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:57.406230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.407095 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.407099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:57.407285 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:57.407308 (+ 23us) client_negotiation.cc:769] Sending connection context
0807 02:25:57.407386 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 02:25:57.407490 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":116,"thread_start_us":130,"threads_started":1}
I20260807 02:25:57.407701 23714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:57.402474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:37403 (local address 127.4.221.65:52961)
0807 02:25:57.402773 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:57.402789 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:57.402967 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:57.403340 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:57.403343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:57.403363 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:57.403539 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:57.403549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.404654 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.404657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:57.406164 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:57.406177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.407095 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.407099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:57.407219 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:57.407277 (+ 58us) client_negotiation.cc:769] Sending connection context
0807 02:25:57.407374 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 02:25:57.407490 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":42,"thread_start_us":109,"threads_started":1}
I20260807 02:25:57.408439 23498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6e6ced51e3445aab39ead06f01049d" candidate_uuid: "252e3b9ad7c5473484490867689acc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ed2b7b3ae84f809b06ffe0e909904b" is_pre_election: true
I20260807 02:25:57.408504 23637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6e6ced51e3445aab39ead06f01049d" candidate_uuid: "252e3b9ad7c5473484490867689acc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ad44a954ee4ce6a5c514450ada6919" is_pre_election: true
I20260807 02:25:57.408792 23498 raft_consensus.cc:2454] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 252e3b9ad7c5473484490867689acc9c in term 0.
I20260807 02:25:57.408798 23637 raft_consensus.cc:2454] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 252e3b9ad7c5473484490867689acc9c in term 0.
I20260807 02:25:57.409318 23286 leader_election.cc:302] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 252e3b9ad7c5473484490867689acc9c, 51ed2b7b3ae84f809b06ffe0e909904b; no voters:
I20260807 02:25:57.409592 23711 raft_consensus.cc:2781] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:25:57.409687 23711 raft_consensus.cc:482] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:25:57.409744 23711 raft_consensus.cc:3037] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:25:57.411067 23711 raft_consensus.cc:504] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ad44a954ee4ce6a5c514450ada6919" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37403 } } peers { permanent_uuid: "252e3b9ad7c5473484490867689acc9c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38783 } } peers { permanent_uuid: "51ed2b7b3ae84f809b06ffe0e909904b" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40533 } }
I20260807 02:25:57.411497 23711 leader_election.cc:288] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [CANDIDATE]: Term 1 election: Requested vote from peers 08ad44a954ee4ce6a5c514450ada6919 (127.4.221.67:37403), 51ed2b7b3ae84f809b06ffe0e909904b (127.4.221.66:40533)
I20260807 02:25:57.411856 23498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6e6ced51e3445aab39ead06f01049d" candidate_uuid: "252e3b9ad7c5473484490867689acc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51ed2b7b3ae84f809b06ffe0e909904b"
I20260807 02:25:57.411856 23637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6e6ced51e3445aab39ead06f01049d" candidate_uuid: "252e3b9ad7c5473484490867689acc9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ad44a954ee4ce6a5c514450ada6919"
I20260807 02:25:57.412025 23498 raft_consensus.cc:3037] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:25:57.411998 23637 raft_consensus.cc:3037] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:25:57.413445 23637 raft_consensus.cc:2454] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 252e3b9ad7c5473484490867689acc9c in term 1.
I20260807 02:25:57.413445 23498 raft_consensus.cc:2454] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 252e3b9ad7c5473484490867689acc9c in term 1.
I20260807 02:25:57.413791 23289 leader_election.cc:302] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ad44a954ee4ce6a5c514450ada6919, 252e3b9ad7c5473484490867689acc9c; no voters:
I20260807 02:25:57.414124 23711 raft_consensus.cc:2781] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:25:57.414376 23711 raft_consensus.cc:686] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [term 1 LEADER]: Becoming Leader. State: Replica: 252e3b9ad7c5473484490867689acc9c, State: Running, Role: LEADER
I20260807 02:25:57.414738 23711 consensus_queue.cc:237] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ad44a954ee4ce6a5c514450ada6919" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37403 } } peers { permanent_uuid: "252e3b9ad7c5473484490867689acc9c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38783 } } peers { permanent_uuid: "51ed2b7b3ae84f809b06ffe0e909904b" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40533 } }
I20260807 02:25:57.418033 23208 catalog_manager.cc:5495] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 252e3b9ad7c5473484490867689acc9c (127.4.221.65). New cstate: current_term: 1 leader_uuid: "252e3b9ad7c5473484490867689acc9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08ad44a954ee4ce6a5c514450ada6919" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "252e3b9ad7c5473484490867689acc9c" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 38783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51ed2b7b3ae84f809b06ffe0e909904b" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40533 } health_report { overall_health: UNKNOWN } } }
W20260807 02:25:57.418552 23687 tablet.cc:2369] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:25:57.458246 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:57.454693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.4.221.65:38783)
0807 02:25:57.454854 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:57.454860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:57.454881 (+ 21us) server_negotiation.cc:407] Connection header received
0807 02:25:57.454958 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:57.454960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:57.455015 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:57.455137 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:25:57.455386 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.455887 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.456750 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:57.456964 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:57.457077 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:57.457382 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:25:57.457790 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:25:57.457958 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:57.458020 (+ 62us) server_negotiation.cc:299] Negotiation successful
0807 02:25:57.458080 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260807 02:25:57.458534 23547 tablet.cc:2369] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:25:57.464186 23637 raft_consensus.cc:1261] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919 [term 1 FOLLOWER]: Refusing update from remote peer 252e3b9ad7c5473484490867689acc9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:25:57.464212 23498 raft_consensus.cc:1261] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b [term 1 FOLLOWER]: Refusing update from remote peer 252e3b9ad7c5473484490867689acc9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:25:57.464874 23711 consensus_queue.cc:1035] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "51ed2b7b3ae84f809b06ffe0e909904b" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:25:57.465119 23716 consensus_queue.cc:1035] T 6b6e6ced51e3445aab39ead06f01049d P 252e3b9ad7c5473484490867689acc9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "08ad44a954ee4ce6a5c514450ada6919" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 37403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:25:57.474028 23720 mvcc.cc:205] Tried to move back new op lower bound from 7315740907366383616 to 7315740907173892096. Current Snapshot: MvccSnapshot[applied={T|T < 7315740907366383616}]
I20260807 02:25:57.480515 23206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:25:57.480695 23206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:25:57.483180 23206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0346130b74e94684a039c9eabf6c5a7d: Sending DeleteTablet for 3 replicas of tablet 6b6e6ced51e3445aab39ead06f01049d
I20260807 02:25:57.483942 23617 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b6e6ced51e3445aab39ead06f01049d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:25:57 UTC) from {username='slave'} at 127.0.0.1:56264
I20260807 02:25:57.483971 23337 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b6e6ced51e3445aab39ead06f01049d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:25:57 UTC) from {username='slave'} at 127.0.0.1:55050
I20260807 02:25:57.484081 23477 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b6e6ced51e3445aab39ead06f01049d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:25:57 UTC) from {username='slave'} at 127.0.0.1:38386
I20260807 02:25:57.484548 23731 tablet_replica.cc:330] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b: stopping tablet replica
I20260807 02:25:57.484879 23731 raft_consensus.cc:2229] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:25:57.485196 23731 raft_consensus.cc:2258] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:25:57.485261 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 23270
I20260807 02:25:57.487049 23731 ts_tablet_manager.cc:1905] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:25:57.490154 23729 tablet_replica.cc:330] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919: stopping tablet replica
I20260807 02:25:57.490504 23729 raft_consensus.cc:2229] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:25:57.490619 23731 ts_tablet_manager.cc:1918] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:25:57.490731 23731 log.cc:1196] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/6b6e6ced51e3445aab39ead06f01049d
I20260807 02:25:57.490897 23729 raft_consensus.cc:2258] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:25:57.491072 23731 ts_tablet_manager.cc:1939] T 6b6e6ced51e3445aab39ead06f01049d P 51ed2b7b3ae84f809b06ffe0e909904b: Deleting consensus metadata
I20260807 02:25:57.492326 23729 ts_tablet_manager.cc:1905] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:25:57.492503 23192 catalog_manager.cc:4841] TS 51ed2b7b3ae84f809b06ffe0e909904b (127.4.221.66:40533): tablet 6b6e6ced51e3445aab39ead06f01049d (table test-table [id=8881a17df0ce4666bbd6a4453e4e8333]) successfully deleted
W20260807 02:25:57.494254 23193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:38783 (error 108)
I20260807 02:25:57.494493 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 23409
I20260807 02:25:57.496047 23729 ts_tablet_manager.cc:1918] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:25:57.496165 23729 log.cc:1196] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/6b6e6ced51e3445aab39ead06f01049d
I20260807 02:25:57.496450 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:57.496038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:38783 (local address 127.0.0.1:55062)
0807 02:25:57.496252 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:57.496342 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:38783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260807 02:25:57.496537 23729 ts_tablet_manager.cc:1939] T 6b6e6ced51e3445aab39ead06f01049d P 08ad44a954ee4ce6a5c514450ada6919: Deleting consensus metadata
W20260807 02:25:57.496788 23193 catalog_manager.cc:4584] TS 252e3b9ad7c5473484490867689acc9c (127.4.221.65:38783): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b6e6ced51e3445aab39ead06f01049d: Network error: Client connection negotiation failed: client connection to 127.4.221.65:38783: connect: Connection refused (error 111)
I20260807 02:25:57.497728 23195 catalog_manager.cc:4841] TS 08ad44a954ee4ce6a5c514450ada6919 (127.4.221.67:37403): tablet 6b6e6ced51e3445aab39ead06f01049d (table test-table [id=8881a17df0ce4666bbd6a4453e4e8333]) successfully deleted
I20260807 02:25:57.501565 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 23549
I20260807 02:25:57.507655 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 23173
2026-08-07T02:25:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1307 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:25:57 dist-test-slave-69rg krb5kdc[23736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:25:57 dist-test-slave-69rg krb5kdc[23736](info): set up 2 sockets
Aug 07 02:25:57 dist-test-slave-69rg krb5kdc[23736](info): commencing operation
krb5kdc: starting...
W20260807 02:25:58.608965 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:25:58 dist-test-slave-69rg krb5kdc[23736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:25:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:25:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:25:58.737438 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:46215
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:46215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:25:58.849351 23750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:58.849643 23750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:58.849716 23750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:58.853720 23750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:25:58.853812 23750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:58.853871 23750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:58.853926 23750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:25:58.853963 23750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:25:58.859145 23750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:46215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:46215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:58.860224 23750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:58.861231 23750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:58.869809 23756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:58.869916 23757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:58.869985 23750 server_base.cc:1034] running on GCE node
W20260807 02:25:58.869829 23760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:58.870709 23750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:58.871791 23750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:58.872987 23750 hybrid_clock.cc:648] HybridClock initialized: now 1786069558872959 us; error 40 us; skew 500 ppm
Aug 07 02:25:58 dist-test-slave-69rg krb5kdc[23736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069558, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:58.879854 23750 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:25:58.880779 23750 webserver.cc:460] Webserver started at http://127.4.221.126:43265/ using document root <none> and password file <none>
I20260807 02:25:58.881386 23750 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:58.881487 23750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:58.881737 23750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:25:58.883909 23750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "a3af3fef87dc4db2b28a2ddb1363fc2c"
format_stamp: "Formatted at 2026-08-07 02:25:58 on dist-test-slave-69rg"
server_key: "7c0fa5a3179fd14746bf5e813736e00d"
server_key_iv: "87c1222af417356447430d0a5a044011"
server_key_version: "encryptionkey@0"
I20260807 02:25:58.884490 23750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "a3af3fef87dc4db2b28a2ddb1363fc2c"
format_stamp: "Formatted at 2026-08-07 02:25:58 on dist-test-slave-69rg"
server_key: "7c0fa5a3179fd14746bf5e813736e00d"
server_key_iv: "87c1222af417356447430d0a5a044011"
server_key_version: "encryptionkey@0"
I20260807 02:25:58.888216 23750 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260807 02:25:58.890594 23767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:58.891733 23750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260807 02:25:58.891871 23750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "a3af3fef87dc4db2b28a2ddb1363fc2c"
format_stamp: "Formatted at 2026-08-07 02:25:58 on dist-test-slave-69rg"
server_key: "7c0fa5a3179fd14746bf5e813736e00d"
server_key_iv: "87c1222af417356447430d0a5a044011"
server_key_version: "encryptionkey@0"
I20260807 02:25:58.891984 23750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:58.908659 23750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:58.912425 23750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:58.912647 23750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:58.920382 23750 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:46215
I20260807 02:25:58.920408 23821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:46215 every 8 connection(s)
I20260807 02:25:58.921337 23750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:25:58.923756 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 23750
I20260807 02:25:58.923868 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:25:58.924216 4981 external_mini_cluster.cc:1428] Setting key 56258f893db5fb6d6c9574ab1d1cca27
I20260807 02:25:58.924229 23822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:25:58.930562 23822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c: Bootstrap starting.
I20260807 02:25:58.932739 23822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c: Neither blocks nor log segments found. Creating new log.
Aug 07 02:25:58 dist-test-slave-69rg krb5kdc[23736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:25:58.933625 23822 log.cc:824] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:58.935894 23822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c: No bootstrap required, opened a new log
I20260807 02:25:58.938303 23825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:58.925652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.4.221.126:46215)
0807 02:25:58.926132 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:58.926149 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:58.926191 (+ 42us) server_negotiation.cc:407] Connection header received
0807 02:25:58.926850 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:58.926867 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:58.927170 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:58.927492 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:58.928338 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:58.929297 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:58.930227 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:58.930736 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:58.930897 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:58.933491 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:58.933496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:58.933508 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:58.933535 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:58.935393 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:58.935882 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:58.935887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:58.935891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:58.936051 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:58.936378 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:58.936380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:58.936381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:58.936755 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:58.936891 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:58.937192 (+ 301us) server_negotiation.cc:299] Negotiation successful
0807 02:25:58.937458 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":185,"threads_started":1}
I20260807 02:25:58.940181 23822 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3af3fef87dc4db2b28a2ddb1363fc2c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46215 } }
I20260807 02:25:58.940454 23822 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:25:58.940542 23822 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3af3fef87dc4db2b28a2ddb1363fc2c, State: Initialized, Role: FOLLOWER
I20260807 02:25:58.941013 23822 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3af3fef87dc4db2b28a2ddb1363fc2c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46215 } }
I20260807 02:25:58.941139 23822 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:25:58.941249 23822 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:25:58.941355 23822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:25:58.942341 23822 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3af3fef87dc4db2b28a2ddb1363fc2c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46215 } }
I20260807 02:25:58.942677 23822 leader_election.cc:302] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3af3fef87dc4db2b28a2ddb1363fc2c; no voters:
I20260807 02:25:58.942975 23822 leader_election.cc:288] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:25:58.943117 23827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:25:58.943392 23827 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [term 1 LEADER]: Becoming Leader. State: Replica: a3af3fef87dc4db2b28a2ddb1363fc2c, State: Running, Role: LEADER
I20260807 02:25:58.943749 23827 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3af3fef87dc4db2b28a2ddb1363fc2c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46215 } }
I20260807 02:25:58.944248 23822 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:25:58.945575 23829 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3af3fef87dc4db2b28a2ddb1363fc2c. Latest consensus state: current_term: 1 leader_uuid: "a3af3fef87dc4db2b28a2ddb1363fc2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3af3fef87dc4db2b28a2ddb1363fc2c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46215 } } }
I20260807 02:25:58.945574 23828 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3af3fef87dc4db2b28a2ddb1363fc2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3af3fef87dc4db2b28a2ddb1363fc2c" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 46215 } } }
I20260807 02:25:58.945752 23829 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [sys.catalog]: This master's current role is: LEADER
I20260807 02:25:58.945755 23828 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c [sys.catalog]: This master's current role is: LEADER
I20260807 02:25:58.946224 23836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:25:58.949361 23836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:25:58.955698 23836 catalog_manager.cc:1347] Generated new cluster ID: 795fd33602a34ebebcd242355ee4c9f6
I20260807 02:25:58.955767 23836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:25:58.977358 23836 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:25:58.978456 23836 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:25:58.990078 23836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c: Generated new TSK 0
I20260807 02:25:58.990917 23836 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:25:59.026499 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:46215
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:25:59.139269 23848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:59.139516 23848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:59.139557 23848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:59.143611 23848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:59.143659 23848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:59.143762 23848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:25:59.149016 23848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:59.150007 23848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:59.150960 23848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:59.160085 23853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:59.160177 23848 server_base.cc:1034] running on GCE node
W20260807 02:25:59.160112 23855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:59.160112 23857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:59.160686 23848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:59.161273 23848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:59.162433 23848 hybrid_clock.cc:648] HybridClock initialized: now 1786069559162410 us; error 34 us; skew 500 ppm
Aug 07 02:25:59 dist-test-slave-69rg krb5kdc[23736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069559, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:59.165145 23848 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:25:59.165936 23848 webserver.cc:460] Webserver started at http://127.4.221.65:46837/ using document root <none> and password file <none>
I20260807 02:25:59.166414 23848 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:59.166471 23848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:59.166630 23848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:25:59.168465 23848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "d2a72fef79b5421990991c1e17b210f0"
format_stamp: "Formatted at 2026-08-07 02:25:59 on dist-test-slave-69rg"
server_key: "dd089793e3c2da328ee2af4abdeb3710"
server_key_iv: "1b6a4d59337229c11c10c0629ded296b"
server_key_version: "encryptionkey@0"
I20260807 02:25:59.168955 23848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "d2a72fef79b5421990991c1e17b210f0"
format_stamp: "Formatted at 2026-08-07 02:25:59 on dist-test-slave-69rg"
server_key: "dd089793e3c2da328ee2af4abdeb3710"
server_key_iv: "1b6a4d59337229c11c10c0629ded296b"
server_key_version: "encryptionkey@0"
I20260807 02:25:59.172446 23848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260807 02:25:59.174701 23865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:59.175830 23848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260807 02:25:59.175922 23848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "d2a72fef79b5421990991c1e17b210f0"
format_stamp: "Formatted at 2026-08-07 02:25:59 on dist-test-slave-69rg"
server_key: "dd089793e3c2da328ee2af4abdeb3710"
server_key_iv: "1b6a4d59337229c11c10c0629ded296b"
server_key_version: "encryptionkey@0"
I20260807 02:25:59.176002 23848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:59.192916 23848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:59.196009 23848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:59.196209 23848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:59.196841 23848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:25:59.197634 23848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:25:59.197695 23848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:59.197734 23848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:25:59.197749 23848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:59.207024 23848 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:39365
I20260807 02:25:59.207055 23985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:39365 every 8 connection(s)
I20260807 02:25:59.207939 23848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:25:59.213650 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 23848
I20260807 02:25:59.213851 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:25:59.214157 4981 external_mini_cluster.cc:1428] Setting key f722bdb9c9e8f018a4c8856097c11d3a
Aug 07 02:25:59 dist-test-slave-69rg krb5kdc[23736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069559, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:25:59.223242 23825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.209973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:54059 (local address 127.4.221.126:46215)
0807 02:25:59.210151 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:59.210157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:59.211069 (+ 912us) server_negotiation.cc:407] Connection header received
0807 02:25:59.212379 (+ 1310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:59.212383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:59.212432 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:59.212527 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:59.213930 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.214568 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.215793 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.216045 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.216164 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.219164 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:59.219167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:59.219169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:59.219224 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:59.220758 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:59.221423 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:59.221426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:59.221427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:59.221507 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:59.221965 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:59.221968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:59.221970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:59.222178 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:59.222280 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:59.222961 (+ 681us) server_negotiation.cc:299] Negotiation successful
0807 02:25:59.223084 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260807 02:25:59.224390 23988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.210399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46215 (local address 127.4.221.65:54059)
0807 02:25:59.210897 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:59.210942 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:59.212141 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:59.212703 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:59.212709 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:59.213071 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:25:59.213738 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.213762 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.214733 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.214743 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.215659 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.215670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.216210 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.216214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.216384 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.217006 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:25:59.217023 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:25:59.218967 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:25:59.220898 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:59.220907 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:59.220926 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:59.221316 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:59.221612 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:59.221616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:59.221619 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:59.221840 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:59.222307 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:25:59.222317 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:25:59.222662 (+ 345us) client_negotiation.cc:769] Sending connection context
0807 02:25:59.222890 (+ 228us) client_negotiation.cc:240] Negotiation successful
0807 02:25:59.223273 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
I20260807 02:25:59.225766 23986 heartbeater.cc:347] Connected to a master server at 127.4.221.126:46215
I20260807 02:25:59.226070 23986 heartbeater.cc:464] Registering TS with master...
I20260807 02:25:59.226686 23986 heartbeater.cc:511] Master 127.4.221.126:46215 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
I20260807 02:25:59.228327 23786 ts_manager.cc:194] Registered new tserver with Master: d2a72fef79b5421990991c1e17b210f0 (127.4.221.65:39365)
I20260807 02:25:59.229698 23786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:54059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:25:59.247411 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:46215
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:25:59.361739 23991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:59.362010 23991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:59.362059 23991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:59.366138 23991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:59.366191 23991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:59.366282 23991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:25:59.371394 23991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:59.372464 23991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:59.373461 23991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:59.383387 23997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:59.383468 23991 server_base.cc:1034] running on GCE node
W20260807 02:25:59.383391 24000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:59.383664 23998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:59.384057 23991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:59.384723 23991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:59.385897 23991 hybrid_clock.cc:648] HybridClock initialized: now 1786069559385874 us; error 24 us; skew 500 ppm
Aug 07 02:25:59 dist-test-slave-69rg krb5kdc[23736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069559, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:59.389098 23991 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:25:59.390126 23991 webserver.cc:460] Webserver started at http://127.4.221.66:40001/ using document root <none> and password file <none>
I20260807 02:25:59.390690 23991 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:59.390782 23991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:59.391013 23991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:25:59.392990 23991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "be66bf9275e542d3adfa161c3adfc334"
format_stamp: "Formatted at 2026-08-07 02:25:59 on dist-test-slave-69rg"
server_key: "b84ae93d96451700d4f026a29cd5eb4b"
server_key_iv: "efc1f9a5661d1354d73384c7961bd08e"
server_key_version: "encryptionkey@0"
I20260807 02:25:59.393627 23991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "be66bf9275e542d3adfa161c3adfc334"
format_stamp: "Formatted at 2026-08-07 02:25:59 on dist-test-slave-69rg"
server_key: "b84ae93d96451700d4f026a29cd5eb4b"
server_key_iv: "efc1f9a5661d1354d73384c7961bd08e"
server_key_version: "encryptionkey@0"
I20260807 02:25:59.397488 23991 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260807 02:25:59.400090 24007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:59.401245 23991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:25:59.401487 23991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "be66bf9275e542d3adfa161c3adfc334"
format_stamp: "Formatted at 2026-08-07 02:25:59 on dist-test-slave-69rg"
server_key: "b84ae93d96451700d4f026a29cd5eb4b"
server_key_iv: "efc1f9a5661d1354d73384c7961bd08e"
server_key_version: "encryptionkey@0"
I20260807 02:25:59.401604 23991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:59.419063 23991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:59.422556 23991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:59.422773 23991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:59.423380 23991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:25:59.424273 23991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:25:59.424341 23991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:59.424415 23991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:25:59.424459 23991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:59.433337 23991 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:37815
I20260807 02:25:59.433367 24125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:37815 every 8 connection(s)
I20260807 02:25:59.434427 23991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
Aug 07 02:25:59 dist-test-slave-69rg krb5kdc[23736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069559, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:25:59.444556 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 23991
I20260807 02:25:59.444686 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:25:59.445062 4981 external_mini_cluster.cc:1428] Setting key 9260c317bc6f3d2afeda0c88b6ffc161
I20260807 02:25:59.448369 23825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.436382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:51677 (local address 127.4.221.126:46215)
0807 02:25:59.436510 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:59.436521 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:59.437268 (+ 747us) server_negotiation.cc:407] Connection header received
0807 02:25:59.438644 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:59.438647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:59.438707 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:59.438791 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:59.440071 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.440616 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.441738 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.442046 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.442176 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.444530 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:59.444534 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:59.444537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:59.444570 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:59.446375 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:59.446876 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:59.446881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:59.446883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:59.446969 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:59.447304 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:59.447308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:59.447310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:59.447491 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:59.447599 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:59.448072 (+ 473us) server_negotiation.cc:299] Negotiation successful
0807 02:25:59.448194 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260807 02:25:59.449402 24128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.436672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46215 (local address 127.4.221.66:51677)
0807 02:25:59.437103 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:59.437144 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:59.438437 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:59.438965 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:59.438972 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:59.439459 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:25:59.439924 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.439936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.440760 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.440763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.441611 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.441620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.442215 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.442220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.442392 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.442912 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:25:59.442927 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:25:59.444367 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:25:59.446477 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:59.446483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:59.446496 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:59.446769 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:59.447063 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:59.447066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:59.447068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:59.447203 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:59.447598 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:25:59.447604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:25:59.447840 (+ 236us) client_negotiation.cc:769] Sending connection context
0807 02:25:59.448016 (+ 176us) client_negotiation.cc:240] Negotiation successful
0807 02:25:59.448319 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":120,"threads_started":1}
I20260807 02:25:59.450619 24126 heartbeater.cc:347] Connected to a master server at 127.4.221.126:46215
I20260807 02:25:59.450884 24126 heartbeater.cc:464] Registering TS with master...
I20260807 02:25:59.451426 24126 heartbeater.cc:511] Master 127.4.221.126:46215 requested a full tablet report, sending...
I20260807 02:25:59.452579 23786 ts_manager.cc:194] Registered new tserver with Master: be66bf9275e542d3adfa161c3adfc334 (127.4.221.66:37815)
I20260807 02:25:59.453161 23786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:51677
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:25:59.474813 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:46215
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:25:59.583856 24131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:59.584096 24131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:59.584136 24131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:59.587975 24131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:59.588018 24131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:59.588071 24131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:25:59.593106 24131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:59.594018 24131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:59.594906 24131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:25:59.603881 24139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:59.603894 24136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:25:59.603890 24137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:25:59.604208 24131 server_base.cc:1034] running on GCE node
I20260807 02:25:59.604671 24131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:25:59.605300 24131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:25:59.606480 24131 hybrid_clock.cc:648] HybridClock initialized: now 1786069559606456 us; error 50 us; skew 500 ppm
Aug 07 02:25:59 dist-test-slave-69rg krb5kdc[23736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069559, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:25:59.609274 24131 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:25:59.610208 24131 webserver.cc:460] Webserver started at http://127.4.221.67:38533/ using document root <none> and password file <none>
I20260807 02:25:59.610807 24131 fs_manager.cc:359] Metadata directory not provided
I20260807 02:25:59.610900 24131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:25:59.611125 24131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:25:59.613066 24131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "a99fe6eece654681914f033a92757105"
format_stamp: "Formatted at 2026-08-07 02:25:59 on dist-test-slave-69rg"
server_key: "48be92182328732ff06b1c2473844235"
server_key_iv: "45109878e9a31fb8f54d9bac9dc0c616"
server_key_version: "encryptionkey@0"
I20260807 02:25:59.613632 24131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "a99fe6eece654681914f033a92757105"
format_stamp: "Formatted at 2026-08-07 02:25:59 on dist-test-slave-69rg"
server_key: "48be92182328732ff06b1c2473844235"
server_key_iv: "45109878e9a31fb8f54d9bac9dc0c616"
server_key_version: "encryptionkey@0"
I20260807 02:25:59.617156 24131 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260807 02:25:59.619515 24146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:59.620698 24131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 02:25:59.620842 24131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "a99fe6eece654681914f033a92757105"
format_stamp: "Formatted at 2026-08-07 02:25:59 on dist-test-slave-69rg"
server_key: "48be92182328732ff06b1c2473844235"
server_key_iv: "45109878e9a31fb8f54d9bac9dc0c616"
server_key_version: "encryptionkey@0"
I20260807 02:25:59.620915 24131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:25:59.648564 24131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:25:59.651664 24131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:25:59.651821 24131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:25:59.652359 24131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:25:59.653119 24131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:25:59.653167 24131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:59.653206 24131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:25:59.653230 24131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:25:59.661993 24131 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:45983
I20260807 02:25:59.662148 24266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:45983 every 8 connection(s)
I20260807 02:25:59.662897 24131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:25:59.671674 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 24131
I20260807 02:25:59.671838 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:25:59.672187 4981 external_mini_cluster.cc:1428] Setting key 6294b83209025905da41360e59ae681f
Aug 07 02:25:59 dist-test-slave-69rg krb5kdc[23736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069559, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:25:59.676838 23825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.664750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:42797 (local address 127.4.221.126:46215)
0807 02:25:59.664889 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:59.664892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:59.665775 (+ 883us) server_negotiation.cc:407] Connection header received
0807 02:25:59.667145 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:59.667149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:59.667204 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:59.667332 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:59.668441 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.668972 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.670189 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.670416 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.670528 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.673123 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:59.673126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:59.673128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:59.673150 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:59.674793 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:59.675376 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:59.675381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:59.675383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:59.675468 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:59.675808 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:59.675813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:59.675814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:59.675991 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:59.676100 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:59.676562 (+ 462us) server_negotiation.cc:299] Negotiation successful
0807 02:25:59.676678 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 02:25:59.677609 24269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.665130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:46215 (local address 127.4.221.67:42797)
0807 02:25:59.665629 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:59.665672 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:59.666940 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:59.667504 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:59.667510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:59.667857 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:25:59.668300 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.668312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.669101 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.669105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.670065 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.670075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.670569 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.670573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.670760 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.671333 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:25:59.671349 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:25:59.672958 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:25:59.674913 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:59.674919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:59.674931 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:59.675260 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:59.675562 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:25:59.675567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:25:59.675569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:25:59.675706 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:25:59.676099 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:25:59.676106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:25:59.676337 (+ 231us) client_negotiation.cc:769] Sending connection context
0807 02:25:59.676520 (+ 183us) client_negotiation.cc:240] Negotiation successful
0807 02:25:59.676830 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":125,"threads_started":1}
I20260807 02:25:59.678673 24267 heartbeater.cc:347] Connected to a master server at 127.4.221.126:46215
I20260807 02:25:59.678906 24267 heartbeater.cc:464] Registering TS with master...
I20260807 02:25:59.679452 24267 heartbeater.cc:511] Master 127.4.221.126:46215 requested a full tablet report, sending...
I20260807 02:25:59.680418 23786 ts_manager.cc:194] Registered new tserver with Master: a99fe6eece654681914f033a92757105 (127.4.221.67:45983)
I20260807 02:25:59.680968 23786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:42797
I20260807 02:25:59.686305 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:25:59.695580 23825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.688189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.4.221.126:46215)
0807 02:25:59.688328 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:59.688333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:59.688522 (+ 189us) server_negotiation.cc:407] Connection header received
0807 02:25:59.688609 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:59.688611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:59.688646 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:59.688710 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:25:59.689432 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.689976 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.690813 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.691011 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.691082 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.692511 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:25:59.692513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:25:59.692515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:25:59.692536 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:25:59.693980 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:59.694413 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:59.694415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:59.694416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:59.694488 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:25:59.694754 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:25:59.694756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:25:59.694758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:25:59.694925 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:25:59.695033 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:59.695319 (+ 286us) server_negotiation.cc:299] Negotiation successful
0807 02:25:59.695414 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 02:25:59.698611 23786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:25:59.701766 23786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:25:59.716833 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.710271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:37815 (local address 127.0.0.1:47262)
0807 02:25:59.711364 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:59.711377 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:59.711471 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:59.712119 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:59.712163 (+ 44us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:59.712175 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:59.712382 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.712390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.713946 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.713949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.715268 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.715277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.716258 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.716262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.716375 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.716420 (+ 45us) client_negotiation.cc:769] Sending connection context
0807 02:25:59.716537 (+ 117us) client_negotiation.cc:240] Negotiation successful
0807 02:25:59.716656 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":210,"thread_start_us":112,"threads_started":1}
I20260807 02:25:59.716850 24283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.710984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.4.221.66:37815)
0807 02:25:59.711621 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:59.711633 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:59.711656 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:25:59.711744 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:59.711749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:59.711949 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:59.712111 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:59.712535 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.713761 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.715394 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.716122 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.716245 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.716394 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:59.716585 (+ 191us) server_negotiation.cc:299] Negotiation successful
0807 02:25:59.716702 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":136,"threads_started":1}
I20260807 02:25:59.717331 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.710544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:39365 (local address 127.0.0.1:55320)
0807 02:25:59.712348 (+ 1804us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:59.712359 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:59.712465 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:59.712870 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:59.712874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:59.712891 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:59.713009 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.713017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.714150 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.714153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.715645 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.715657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.716902 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.716908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.717016 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.717042 (+ 26us) client_negotiation.cc:769] Sending connection context
0807 02:25:59.717120 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 02:25:59.717198 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1753,"mutex_wait_us":100,"thread_start_us":93,"threads_started":1}
I20260807 02:25:59.717614 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.710211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:45983 (local address 127.0.0.1:35282)
0807 02:25:59.710930 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:59.710981 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:59.711110 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:59.712119 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:59.712125 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:59.712151 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:59.712382 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.712390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.713946 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.713949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.715653 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.715659 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.717224 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.717228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.717321 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.717348 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 02:25:59.717425 (+ 77us) client_negotiation.cc:240] Negotiation successful
0807 02:25:59.717498 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":424,"thread_start_us":86,"threads_started":1}
I20260807 02:25:59.717759 24282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.710984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.4.221.65:39365)
0807 02:25:59.711417 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:59.711435 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:59.712456 (+ 1021us) server_negotiation.cc:407] Connection header received
0807 02:25:59.712597 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:59.712601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:59.712736 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:59.712889 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:59.713135 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.713990 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.715785 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.716587 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.716696 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.716829 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:59.717494 (+ 665us) server_negotiation.cc:299] Negotiation successful
0807 02:25:59.717588 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":74,"threads_started":1}
I20260807 02:25:59.718039 24280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.710303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35282 (local address 127.4.221.67:45983)
0807 02:25:59.711714 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:59.711726 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:59.711744 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:25:59.711807 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:59.711811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:59.711949 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:59.712111 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:59.712535 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.713761 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.715796 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.716448 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.716560 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.716703 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:59.717777 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0807 02:25:59.717908 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1312,"thread_start_us":156,"threads_started":1}
I20260807 02:25:59.720100 24060 tablet_service.cc:1467] Processing CreateTablet for tablet edc249c4d88643ba8d415631faddaeab (DEFAULT_TABLE table=test-table [id=3425e235fc9e490f8cb9f91832aae515]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:25:59.720619 24197 tablet_service.cc:1467] Processing CreateTablet for tablet edc249c4d88643ba8d415631faddaeab (DEFAULT_TABLE table=test-table [id=3425e235fc9e490f8cb9f91832aae515]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:25:59.721298 24060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc249c4d88643ba8d415631faddaeab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:25:59.721607 24197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc249c4d88643ba8d415631faddaeab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:25:59.724491 23918 tablet_service.cc:1467] Processing CreateTablet for tablet edc249c4d88643ba8d415631faddaeab (DEFAULT_TABLE table=test-table [id=3425e235fc9e490f8cb9f91832aae515]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:25:59.725533 23918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc249c4d88643ba8d415631faddaeab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:25:59.728708 24284 tablet_bootstrap.cc:492] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334: Bootstrap starting.
I20260807 02:25:59.729604 24285 tablet_bootstrap.cc:492] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105: Bootstrap starting.
I20260807 02:25:59.731887 24284 tablet_bootstrap.cc:654] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334: Neither blocks nor log segments found. Creating new log.
I20260807 02:25:59.731951 24285 tablet_bootstrap.cc:654] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105: Neither blocks nor log segments found. Creating new log.
I20260807 02:25:59.732589 24284 log.cc:824] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:59.732872 24286 tablet_bootstrap.cc:492] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0: Bootstrap starting.
I20260807 02:25:59.734201 24285 log.cc:824] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:59.735080 24286 tablet_bootstrap.cc:654] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0: Neither blocks nor log segments found. Creating new log.
I20260807 02:25:59.735958 24286 log.cc:824] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0: Log is configured to *not* fsync() on all Append() calls
I20260807 02:25:59.736744 24285 tablet_bootstrap.cc:492] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105: No bootstrap required, opened a new log
I20260807 02:25:59.736986 24284 tablet_bootstrap.cc:492] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334: No bootstrap required, opened a new log
I20260807 02:25:59.737011 24285 ts_tablet_manager.cc:1397] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260807 02:25:59.737188 24284 ts_tablet_manager.cc:1397] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260807 02:25:59.738131 24286 tablet_bootstrap.cc:492] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0: No bootstrap required, opened a new log
I20260807 02:25:59.738302 24286 ts_tablet_manager.cc:1397] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260807 02:25:59.740222 24284 raft_consensus.cc:348] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fe6eece654681914f033a92757105" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45983 } } peers { permanent_uuid: "be66bf9275e542d3adfa161c3adfc334" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 37815 } } peers { permanent_uuid: "d2a72fef79b5421990991c1e17b210f0" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39365 } }
I20260807 02:25:59.740635 24284 raft_consensus.cc:374] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:25:59.740756 24284 raft_consensus.cc:729] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be66bf9275e542d3adfa161c3adfc334, State: Initialized, Role: FOLLOWER
I20260807 02:25:59.741143 24285 raft_consensus.cc:348] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fe6eece654681914f033a92757105" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45983 } } peers { permanent_uuid: "be66bf9275e542d3adfa161c3adfc334" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 37815 } } peers { permanent_uuid: "d2a72fef79b5421990991c1e17b210f0" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39365 } }
I20260807 02:25:59.741330 24284 consensus_queue.cc:260] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fe6eece654681914f033a92757105" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45983 } } peers { permanent_uuid: "be66bf9275e542d3adfa161c3adfc334" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 37815 } } peers { permanent_uuid: "d2a72fef79b5421990991c1e17b210f0" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39365 } }
I20260807 02:25:59.741547 24285 raft_consensus.cc:374] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:25:59.741644 24285 raft_consensus.cc:729] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a99fe6eece654681914f033a92757105, State: Initialized, Role: FOLLOWER
I20260807 02:25:59.741961 24126 heartbeater.cc:503] Master 127.4.221.126:46215 was elected leader, sending a full tablet report...
I20260807 02:25:59.742149 24285 consensus_queue.cc:260] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fe6eece654681914f033a92757105" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45983 } } peers { permanent_uuid: "be66bf9275e542d3adfa161c3adfc334" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 37815 } } peers { permanent_uuid: "d2a72fef79b5421990991c1e17b210f0" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39365 } }
I20260807 02:25:59.742326 24284 ts_tablet_manager.cc:1428] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 02:25:59.742600 24286 raft_consensus.cc:348] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fe6eece654681914f033a92757105" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45983 } } peers { permanent_uuid: "be66bf9275e542d3adfa161c3adfc334" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 37815 } } peers { permanent_uuid: "d2a72fef79b5421990991c1e17b210f0" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39365 } }
I20260807 02:25:59.742990 24267 heartbeater.cc:503] Master 127.4.221.126:46215 was elected leader, sending a full tablet report...
I20260807 02:25:59.743058 24285 ts_tablet_manager.cc:1428] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260807 02:25:59.743782 24286 raft_consensus.cc:374] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:25:59.743856 24286 raft_consensus.cc:729] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2a72fef79b5421990991c1e17b210f0, State: Initialized, Role: FOLLOWER
I20260807 02:25:59.744306 24286 consensus_queue.cc:260] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fe6eece654681914f033a92757105" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45983 } } peers { permanent_uuid: "be66bf9275e542d3adfa161c3adfc334" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 37815 } } peers { permanent_uuid: "d2a72fef79b5421990991c1e17b210f0" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39365 } }
I20260807 02:25:59.745008 23986 heartbeater.cc:503] Master 127.4.221.126:46215 was elected leader, sending a full tablet report...
I20260807 02:25:59.745455 24286 ts_tablet_manager.cc:1428] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260807 02:25:59.751315 24292 raft_consensus.cc:482] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:25:59.751451 24292 raft_consensus.cc:504] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fe6eece654681914f033a92757105" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45983 } } peers { permanent_uuid: "be66bf9275e542d3adfa161c3adfc334" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 37815 } } peers { permanent_uuid: "d2a72fef79b5421990991c1e17b210f0" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39365 } }
I20260807 02:25:59.752355 24292 leader_election.cc:288] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be66bf9275e542d3adfa161c3adfc334 (127.4.221.66:37815), d2a72fef79b5421990991c1e17b210f0 (127.4.221.65:39365)
I20260807 02:25:59.755863 24269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.752607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:39365 (local address 127.4.221.67:52001)
0807 02:25:59.752732 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:59.752749 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:59.752839 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:59.753090 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:59.753093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:59.753108 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:59.753244 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.753249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.754018 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.754023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.754785 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.754794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.755441 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.755445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.755547 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.755574 (+ 27us) client_negotiation.cc:769] Sending connection context
0807 02:25:59.755662 (+ 88us) client_negotiation.cc:240] Negotiation successful
0807 02:25:59.755735 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260807 02:25:59.755863 24282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.752703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:52001 (local address 127.4.221.65:39365)
0807 02:25:59.752821 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:59.752826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:59.752839 (+ 13us) server_negotiation.cc:407] Connection header received
0807 02:25:59.752949 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:59.752952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:59.753000 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:59.753080 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:59.753353 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.753916 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.754923 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.755323 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.755449 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.755500 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:59.755676 (+ 176us) server_negotiation.cc:299] Negotiation successful
0807 02:25:59.755748 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 02:25:59.756248 24283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.752712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:55029 (local address 127.4.221.66:37815)
0807 02:25:59.752842 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:59.752849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:59.753125 (+ 276us) server_negotiation.cc:407] Connection header received
0807 02:25:59.753277 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:59.753280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:59.753323 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:59.753414 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:25:59.753686 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.754236 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.755266 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.755684 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.755809 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.755855 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:59.756052 (+ 197us) server_negotiation.cc:299] Negotiation successful
0807 02:25:59.756122 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 02:25:59.756546 23939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc249c4d88643ba8d415631faddaeab" candidate_uuid: "a99fe6eece654681914f033a92757105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2a72fef79b5421990991c1e17b210f0" is_pre_election: true
I20260807 02:25:59.756827 23939 raft_consensus.cc:2454] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a99fe6eece654681914f033a92757105 in term 0.
I20260807 02:25:59.756276 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.752607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:37815 (local address 127.4.221.67:55029)
0807 02:25:59.753028 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:59.753040 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0807 02:25:59.753164 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:25:59.753421 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:25:59.753424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:25:59.753438 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:25:59.753564 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.753570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.754341 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.754347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.755108 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:25:59.755116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.755809 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.755813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:25:59.755917 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.755949 (+ 32us) client_negotiation.cc:769] Sending connection context
0807 02:25:59.756033 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 02:25:59.756104 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":120,"thread_start_us":120,"threads_started":1}
I20260807 02:25:59.757440 24147 leader_election.cc:302] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a99fe6eece654681914f033a92757105, d2a72fef79b5421990991c1e17b210f0; no voters:
I20260807 02:25:59.757618 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc249c4d88643ba8d415631faddaeab" candidate_uuid: "a99fe6eece654681914f033a92757105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be66bf9275e542d3adfa161c3adfc334" is_pre_election: true
I20260807 02:25:59.757701 24292 raft_consensus.cc:2781] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:25:59.757794 24292 raft_consensus.cc:482] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:25:59.757875 24292 raft_consensus.cc:3037] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:25:59.757916 24080 raft_consensus.cc:2454] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a99fe6eece654681914f033a92757105 in term 0.
I20260807 02:25:59.758947 24292 raft_consensus.cc:504] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fe6eece654681914f033a92757105" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45983 } } peers { permanent_uuid: "be66bf9275e542d3adfa161c3adfc334" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 37815 } } peers { permanent_uuid: "d2a72fef79b5421990991c1e17b210f0" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39365 } }
I20260807 02:25:59.759311 24292 leader_election.cc:288] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [CANDIDATE]: Term 1 election: Requested vote from peers be66bf9275e542d3adfa161c3adfc334 (127.4.221.66:37815), d2a72fef79b5421990991c1e17b210f0 (127.4.221.65:39365)
I20260807 02:25:59.759575 24080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc249c4d88643ba8d415631faddaeab" candidate_uuid: "a99fe6eece654681914f033a92757105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be66bf9275e542d3adfa161c3adfc334"
I20260807 02:25:59.759652 23939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc249c4d88643ba8d415631faddaeab" candidate_uuid: "a99fe6eece654681914f033a92757105" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2a72fef79b5421990991c1e17b210f0"
I20260807 02:25:59.759727 24080 raft_consensus.cc:3037] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:25:59.759794 23939 raft_consensus.cc:3037] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:25:59.760725 23939 raft_consensus.cc:2454] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a99fe6eece654681914f033a92757105 in term 1.
I20260807 02:25:59.760874 24080 raft_consensus.cc:2454] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a99fe6eece654681914f033a92757105 in term 1.
I20260807 02:25:59.760995 24147 leader_election.cc:302] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a99fe6eece654681914f033a92757105, d2a72fef79b5421990991c1e17b210f0; no voters:
I20260807 02:25:59.761183 24292 raft_consensus.cc:2781] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:25:59.761412 24292 raft_consensus.cc:686] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [term 1 LEADER]: Becoming Leader. State: Replica: a99fe6eece654681914f033a92757105, State: Running, Role: LEADER
I20260807 02:25:59.761740 24292 consensus_queue.cc:237] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fe6eece654681914f033a92757105" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45983 } } peers { permanent_uuid: "be66bf9275e542d3adfa161c3adfc334" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 37815 } } peers { permanent_uuid: "d2a72fef79b5421990991c1e17b210f0" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39365 } }
I20260807 02:25:59.764631 23784 catalog_manager.cc:5495] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 reported cstate change: term changed from 0 to 1, leader changed from <none> to a99fe6eece654681914f033a92757105 (127.4.221.67). New cstate: current_term: 1 leader_uuid: "a99fe6eece654681914f033a92757105" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a99fe6eece654681914f033a92757105" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be66bf9275e542d3adfa161c3adfc334" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 37815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2a72fef79b5421990991c1e17b210f0" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39365 } health_report { overall_health: UNKNOWN } } }
I20260807 02:25:59.781694 24280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.778181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.4.221.67:45983)
0807 02:25:59.778305 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 02:25:59.778309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:25:59.778322 (+ 13us) server_negotiation.cc:407] Connection header received
0807 02:25:59.778444 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:25:59.778446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:25:59.778497 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:25:59.778580 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:25:59.778795 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.779271 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.780196 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:25:59.780411 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:25:59.780539 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:25:59.780868 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:25:59.781245 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:25:59.781400 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 02:25:59.781492 (+ 92us) server_negotiation.cc:299] Negotiation successful
0807 02:25:59.781529 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260807 02:25:59.787250 24080 raft_consensus.cc:1261] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334 [term 1 FOLLOWER]: Refusing update from remote peer a99fe6eece654681914f033a92757105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:25:59.787249 23939 raft_consensus.cc:1261] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0 [term 1 FOLLOWER]: Refusing update from remote peer a99fe6eece654681914f033a92757105: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:25:59.787922 24292 consensus_queue.cc:1035] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2a72fef79b5421990991c1e17b210f0" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 39365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:25:59.788098 24296 consensus_queue.cc:1035] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be66bf9275e542d3adfa161c3adfc334" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 37815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:25:59.796728 24301 mvcc.cc:205] Tried to move back new op lower bound from 7315740916881604608 to 7315740916786688000. Current Snapshot: MvccSnapshot[applied={T|T < 7315740916881604608}]
I20260807 02:25:59.798595 24300 mvcc.cc:205] Tried to move back new op lower bound from 7315740916881604608 to 7315740916786688000. Current Snapshot: MvccSnapshot[applied={T|T < 7315740916881604608}]
I20260807 02:25:59.804005 23786 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:25:59.804191 23786 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:25:59.806558 23786 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3af3fef87dc4db2b28a2ddb1363fc2c: Sending DeleteTablet for 3 replicas of tablet edc249c4d88643ba8d415631faddaeab
I20260807 02:25:59.807322 24197 tablet_service.cc:1514] Processing DeleteTablet for tablet edc249c4d88643ba8d415631faddaeab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:25:59 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:35282
I20260807 02:25:59.807363 24060 tablet_service.cc:1514] Processing DeleteTablet for tablet edc249c4d88643ba8d415631faddaeab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:25:59 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:47262
I20260807 02:25:59.807816 24309 tablet_replica.cc:330] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105: stopping tablet replica
I20260807 02:25:59.807852 23918 tablet_service.cc:1514] Processing DeleteTablet for tablet edc249c4d88643ba8d415631faddaeab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:25:59 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:55320
I20260807 02:25:59.808125 24309 raft_consensus.cc:2229] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:25:59.808297 24311 tablet_replica.cc:330] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0: stopping tablet replica
I20260807 02:25:59.808509 24309 raft_consensus.cc:2258] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:25:59.808580 24311 raft_consensus.cc:2229] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:25:59.808904 24311 raft_consensus.cc:2258] T edc249c4d88643ba8d415631faddaeab P d2a72fef79b5421990991c1e17b210f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:25:59.809108 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 23848
I20260807 02:25:59.813612 24310 tablet_replica.cc:330] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334: stopping tablet replica
I20260807 02:25:59.813930 24310 raft_consensus.cc:2229] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:25:59.814205 24310 raft_consensus.cc:2258] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:25:59.815807 24310 ts_tablet_manager.cc:1905] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:25:59.816392 24309 ts_tablet_manager.cc:1905] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:25:59.819046 24309 ts_tablet_manager.cc:1918] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:25:59.819180 24309 log.cc:1196] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/edc249c4d88643ba8d415631faddaeab
I20260807 02:25:59.819535 24309 ts_tablet_manager.cc:1939] T edc249c4d88643ba8d415631faddaeab P a99fe6eece654681914f033a92757105: Deleting consensus metadata
I20260807 02:25:59.819523 24310 ts_tablet_manager.cc:1918] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:25:59.819617 24310 log.cc:1196] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/edc249c4d88643ba8d415631faddaeab
I20260807 02:25:59.819998 24310 ts_tablet_manager.cc:1939] T edc249c4d88643ba8d415631faddaeab P be66bf9275e542d3adfa161c3adfc334: Deleting consensus metadata
W20260807 02:25:59.820732 23768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:39365 (error 108)
I20260807 02:25:59.821159 23769 catalog_manager.cc:4841] TS be66bf9275e542d3adfa161c3adfc334 (127.4.221.66:37815): tablet edc249c4d88643ba8d415631faddaeab (table test-table [id=3425e235fc9e490f8cb9f91832aae515]) successfully deleted
I20260807 02:25:59.821286 23771 catalog_manager.cc:4841] TS a99fe6eece654681914f033a92757105 (127.4.221.67:45983): tablet edc249c4d88643ba8d415631faddaeab (table test-table [id=3425e235fc9e490f8cb9f91832aae515]) successfully deleted
I20260807 02:25:59.821892 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 23991
I20260807 02:25:59.822083 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:25:59.821777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:39365 (local address 127.0.0.1:55328)
0807 02:25:59.821912 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0807 02:25:59.821986 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:39365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260807 02:25:59.822794 23768 catalog_manager.cc:4584] TS d2a72fef79b5421990991c1e17b210f0 (127.4.221.65:39365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet edc249c4d88643ba8d415631faddaeab: Network error: Client connection negotiation failed: client connection to 127.4.221.65:39365: connect: Connection refused (error 111)
I20260807 02:25:59.830471 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 24131
I20260807 02:25:59.837478 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 23750
2026-08-07T02:25:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2330 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-07T02:25:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:25:59Z Disabled control of system clock
I20260807 02:25:59.870884 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:37351
--webserver_interface=127.4.221.126
--webserver_port=0
--builtin_ntp_servers=127.4.221.84:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:37351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 02:25:59.983562 24316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:25:59.983839 24316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:25:59.983877 24316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:25:59.987766 24316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:25:59.987819 24316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:25:59.987835 24316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:25:59.987851 24316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:25:59.987867 24316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:25:59.992789 24316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:37351
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:37351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:25:59.993858 24316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:25:59.995190 24316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:00.003708 24325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:00.003741 24322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:00.003710 24323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:00.003790 24316 server_base.cc:1034] running on GCE node
I20260807 02:26:00.004470 24316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:00.005941 24316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:00.007122 24316 hybrid_clock.cc:648] HybridClock initialized: now 1786069560007095 us; error 38 us; skew 500 ppm
I20260807 02:26:00.009025 24316 webserver.cc:460] Webserver started at http://127.4.221.126:34027/ using document root <none> and password file <none>
I20260807 02:26:00.009613 24316 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:00.009675 24316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:00.009960 24316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:00.011972 24316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "70aa3c17017e45fa8afcafb64e5befbd"
format_stamp: "Formatted at 2026-08-07 02:26:00 on dist-test-slave-69rg"
server_key: "fcde1763a3a779f40498d2f639f4e8bb"
server_key_iv: "946ddb15be465c258ccb5ba435575158"
server_key_version: "encryptionkey@0"
I20260807 02:26:00.012553 24316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "70aa3c17017e45fa8afcafb64e5befbd"
format_stamp: "Formatted at 2026-08-07 02:26:00 on dist-test-slave-69rg"
server_key: "fcde1763a3a779f40498d2f639f4e8bb"
server_key_iv: "946ddb15be465c258ccb5ba435575158"
server_key_version: "encryptionkey@0"
I20260807 02:26:00.016317 24316 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 02:26:00.018790 24332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:00.020001 24316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:26:00.020165 24316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "70aa3c17017e45fa8afcafb64e5befbd"
format_stamp: "Formatted at 2026-08-07 02:26:00 on dist-test-slave-69rg"
server_key: "fcde1763a3a779f40498d2f639f4e8bb"
server_key_iv: "946ddb15be465c258ccb5ba435575158"
server_key_version: "encryptionkey@0"
I20260807 02:26:00.020287 24316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:00.040092 24316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:00.040943 24316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:00.041201 24316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:00.048563 24316 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:37351
I20260807 02:26:00.048568 24386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:37351 every 8 connection(s)
I20260807 02:26:00.049616 24316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:00.052665 24387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:00.057050 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 24316
I20260807 02:26:00.057196 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:00.057487 4981 external_mini_cluster.cc:1428] Setting key d6f43d49898d53de2eb2f8dc13dec291
I20260807 02:26:00.058285 24387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd: Bootstrap starting.
I20260807 02:26:00.060987 24387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:00.061893 24387 log.cc:824] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:00.064397 24387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd: No bootstrap required, opened a new log
I20260807 02:26:00.069128 24387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa3c17017e45fa8afcafb64e5befbd" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37351 } }
I20260807 02:26:00.069378 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:00.059074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46590 (local address 127.4.221.126:37351)
0807 02:26:00.059881 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:00.059899 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:00.059942 (+ 43us) server_negotiation.cc:407] Connection header received
0807 02:26:00.060764 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:00.060786 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:00.061310 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:00.061657 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:00.062291 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.063946 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.065191 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.065665 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.065886 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:00.067209 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:00.067215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:00.067239 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:00.067756 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:00.067832 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:00.067863 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:00.067980 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:00.068103 (+ 123us) server_negotiation.cc:299] Negotiation successful
0807 02:26:00.068426 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":149,"threads_started":1}
I20260807 02:26:00.069410 24387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:00.069456 24387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70aa3c17017e45fa8afcafb64e5befbd, State: Initialized, Role: FOLLOWER
I20260807 02:26:00.069942 24387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa3c17017e45fa8afcafb64e5befbd" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37351 } }
I20260807 02:26:00.070072 24387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:00.070156 24387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:00.070273 24387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:00.071441 24387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa3c17017e45fa8afcafb64e5befbd" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37351 } }
I20260807 02:26:00.071791 24387 leader_election.cc:302] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70aa3c17017e45fa8afcafb64e5befbd; no voters:
I20260807 02:26:00.072109 24387 leader_election.cc:288] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:00.072234 24392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:00.072482 24392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [term 1 LEADER]: Becoming Leader. State: Replica: 70aa3c17017e45fa8afcafb64e5befbd, State: Running, Role: LEADER
I20260807 02:26:00.072801 24392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa3c17017e45fa8afcafb64e5befbd" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37351 } }
I20260807 02:26:00.073369 24387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:00.074452 24393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70aa3c17017e45fa8afcafb64e5befbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa3c17017e45fa8afcafb64e5befbd" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37351 } } }
I20260807 02:26:00.074604 24393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:00.074898 24394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70aa3c17017e45fa8afcafb64e5befbd. Latest consensus state: current_term: 1 leader_uuid: "70aa3c17017e45fa8afcafb64e5befbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa3c17017e45fa8afcafb64e5befbd" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37351 } } }
I20260807 02:26:00.074988 24401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:00.074996 24394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:00.078370 24401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:00.085608 24401 catalog_manager.cc:1347] Generated new cluster ID: 7ede5d7dce58493cb3a77c801cbb8fd6
I20260807 02:26:00.085675 24401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:00.097962 24401 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:00.098888 24401 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:00.110507 24401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd: Generated new TSK 0
I20260807 02:26:00.111183 24401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:00.114312 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:37351
--builtin_ntp_servers=127.4.221.84:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 02:26:00.225519 24412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:00.225764 24412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:00.225801 24412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:00.229918 24412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:00.229964 24412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:00.230048 24412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:26:00.234807 24412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:37351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:00.235872 24412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:00.237244 24412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:00.247117 24417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:00.247402 24412 server_base.cc:1034] running on GCE node
W20260807 02:26:00.247152 24418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:00.247117 24420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:00.247900 24412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:00.248574 24412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:00.249768 24412 hybrid_clock.cc:648] HybridClock initialized: now 1786069560249753 us; error 26 us; skew 500 ppm
I20260807 02:26:00.251681 24412 webserver.cc:460] Webserver started at http://127.4.221.65:36301/ using document root <none> and password file <none>
I20260807 02:26:00.252259 24412 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:00.252352 24412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:00.252580 24412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:00.254678 24412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "476905c514f54b1d863aaa8cf7523746"
format_stamp: "Formatted at 2026-08-07 02:26:00 on dist-test-slave-69rg"
server_key: "799d1e4369a45c337c46a36004ea41b2"
server_key_iv: "df40af111b606d4b1473956696344641"
server_key_version: "encryptionkey@0"
I20260807 02:26:00.255266 24412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "476905c514f54b1d863aaa8cf7523746"
format_stamp: "Formatted at 2026-08-07 02:26:00 on dist-test-slave-69rg"
server_key: "799d1e4369a45c337c46a36004ea41b2"
server_key_iv: "df40af111b606d4b1473956696344641"
server_key_version: "encryptionkey@0"
I20260807 02:26:00.259032 24412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:26:00.261565 24427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:00.262933 24412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:26:00.263103 24412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "476905c514f54b1d863aaa8cf7523746"
format_stamp: "Formatted at 2026-08-07 02:26:00 on dist-test-slave-69rg"
server_key: "799d1e4369a45c337c46a36004ea41b2"
server_key_iv: "df40af111b606d4b1473956696344641"
server_key_version: "encryptionkey@0"
I20260807 02:26:00.263240 24412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:00.274279 24412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:00.274955 24412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:00.275157 24412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:00.275717 24412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:26:00.276654 24412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:26:00.276703 24412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:00.276779 24412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:26:00.276814 24412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:00.285740 24412 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:43253
I20260807 02:26:00.285768 24545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:43253 every 8 connection(s)
I20260807 02:26:00.286662 24412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:26:00.291340 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 24412
I20260807 02:26:00.291504 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:26:00.291836 4981 external_mini_cluster.cc:1428] Setting key 53b73469438e7619566c894a2ec06b98
I20260807 02:26:00.294073 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:37351
--builtin_ntp_servers=127.4.221.84:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 02:26:00.297477 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:00.288524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:58301 (local address 127.4.221.126:37351)
0807 02:26:00.288662 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:00.288666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:00.289864 (+ 1198us) server_negotiation.cc:407] Connection header received
0807 02:26:00.291327 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:00.291330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:00.291378 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:00.291492 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:00.292461 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.293257 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.295039 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.295273 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.295358 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:00.296862 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:00.296869 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:00.296873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:00.296893 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:00.296919 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:00.296930 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:00.297016 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:00.297193 (+ 177us) server_negotiation.cc:299] Negotiation successful
0807 02:26:00.297267 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 02:26:00.298379 24548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:00.288912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:37351 (local address 127.4.221.65:58301)
0807 02:26:00.289671 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:00.289711 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:00.291085 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:00.291691 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:00.291697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:00.291778 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:00.292311 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:00.292325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.293416 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.293420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:00.294882 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:00.294897 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.295436 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.295443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:00.295681 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:00.296571 (+ 890us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:26:00.296573 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:00.296614 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:26:00.296685 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:00.297022 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:00.297029 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:00.297033 (+ 4us) client_negotiation.cc:769] Sending connection context
0807 02:26:00.297130 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 02:26:00.297362 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":164,"threads_started":1}
I20260807 02:26:00.299645 24546 heartbeater.cc:347] Connected to a master server at 127.4.221.126:37351
I20260807 02:26:00.299880 24546 heartbeater.cc:464] Registering TS with master...
I20260807 02:26:00.300400 24546 heartbeater.cc:511] Master 127.4.221.126:37351 requested a full tablet report, sending...
I20260807 02:26:00.301942 24350 ts_manager.cc:194] Registered new tserver with Master: 476905c514f54b1d863aaa8cf7523746 (127.4.221.65:43253)
I20260807 02:26:00.303059 24350 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.221.65:58301
W20260807 02:26:00.406996 24549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:00.407245 24549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:00.407284 24549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:00.411342 24549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:00.411393 24549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:00.411478 24549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:26:00.416186 24549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:37351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:00.417302 24549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:00.418748 24549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:00.428602 24556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:00.428562 24555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:00.428560 24558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:00.428987 24549 server_base.cc:1034] running on GCE node
I20260807 02:26:00.429415 24549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:00.430172 24549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:00.431367 24549 hybrid_clock.cc:648] HybridClock initialized: now 1786069560431358 us; error 31 us; skew 500 ppm
I20260807 02:26:00.433306 24549 webserver.cc:460] Webserver started at http://127.4.221.66:42767/ using document root <none> and password file <none>
I20260807 02:26:00.433931 24549 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:00.433998 24549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:00.434228 24549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:00.436233 24549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "781d84fe860e43d0ae7ecfb1d12f44cc"
format_stamp: "Formatted at 2026-08-07 02:26:00 on dist-test-slave-69rg"
server_key: "1ec6424eec0bf20bda8d1f1747ef8aa2"
server_key_iv: "553bcb809e58eac29cc9ac7a067c4c36"
server_key_version: "encryptionkey@0"
I20260807 02:26:00.436779 24549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "781d84fe860e43d0ae7ecfb1d12f44cc"
format_stamp: "Formatted at 2026-08-07 02:26:00 on dist-test-slave-69rg"
server_key: "1ec6424eec0bf20bda8d1f1747ef8aa2"
server_key_iv: "553bcb809e58eac29cc9ac7a067c4c36"
server_key_version: "encryptionkey@0"
I20260807 02:26:00.440352 24549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260807 02:26:00.442739 24564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:00.444011 24549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:26:00.444130 24549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "781d84fe860e43d0ae7ecfb1d12f44cc"
format_stamp: "Formatted at 2026-08-07 02:26:00 on dist-test-slave-69rg"
server_key: "1ec6424eec0bf20bda8d1f1747ef8aa2"
server_key_iv: "553bcb809e58eac29cc9ac7a067c4c36"
server_key_version: "encryptionkey@0"
I20260807 02:26:00.444236 24549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:00.476541 24549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:00.477260 24549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:00.477466 24549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:00.478097 24549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:26:00.479051 24549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:26:00.479097 24549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:00.479166 24549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:26:00.479198 24549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:00.488104 24549 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:40643
I20260807 02:26:00.488152 24682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:40643 every 8 connection(s)
I20260807 02:26:00.488991 24549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:26:00.492200 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 24549
I20260807 02:26:00.492341 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:26:00.492630 4981 external_mini_cluster.cc:1428] Setting key 34ec6864c621d821f0a7353d6dc5a088
I20260807 02:26:00.494639 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:37351
--builtin_ntp_servers=127.4.221.84:45969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 02:26:00.498731 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:00.490810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:53707 (local address 127.4.221.126:37351)
0807 02:26:00.490977 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:00.490981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:00.491985 (+ 1004us) server_negotiation.cc:407] Connection header received
0807 02:26:00.493668 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:00.493671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:00.493723 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:00.493840 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:00.494803 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.495611 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.496743 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.496953 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.497056 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:00.498234 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:00.498237 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:00.498244 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:00.498256 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:00.498275 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:00.498282 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:00.498358 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:00.498519 (+ 161us) server_negotiation.cc:299] Negotiation successful
0807 02:26:00.498581 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260807 02:26:00.499568 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:00.491174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:37351 (local address 127.4.221.66:53707)
0807 02:26:00.491833 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:00.491873 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:00.493466 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:00.494060 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:00.494071 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:00.494169 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:00.494621 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:00.494634 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.495759 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.495762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:00.496600 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:00.496609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.497106 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.497111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:00.497295 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:00.497946 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:26:00.497947 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:00.497984 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:26:00.498042 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:00.498362 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:00.498373 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:00.498380 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 02:26:00.498514 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 02:26:00.498774 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":116,"threads_started":1}
I20260807 02:26:00.500680 24683 heartbeater.cc:347] Connected to a master server at 127.4.221.126:37351
I20260807 02:26:00.500916 24683 heartbeater.cc:464] Registering TS with master...
I20260807 02:26:00.501418 24683 heartbeater.cc:511] Master 127.4.221.126:37351 requested a full tablet report, sending...
I20260807 02:26:00.502517 24350 ts_manager.cc:194] Registered new tserver with Master: 781d84fe860e43d0ae7ecfb1d12f44cc (127.4.221.66:40643)
I20260807 02:26:00.503363 24350 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.221.66:53707
W20260807 02:26:00.608733 24686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:00.609009 24686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:00.609050 24686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:00.613126 24686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:00.613189 24686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:00.613245 24686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:26:00.618013 24686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:45969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:37351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:00.619124 24686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:00.620440 24686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:00.630172 24692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:00.630234 24686 server_base.cc:1034] running on GCE node
W20260807 02:26:00.630178 24693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:00.630246 24695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:00.630803 24686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:00.631430 24686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:00.632625 24686 hybrid_clock.cc:648] HybridClock initialized: now 1786069560632611 us; error 30 us; skew 500 ppm
I20260807 02:26:00.634656 24686 webserver.cc:460] Webserver started at http://127.4.221.67:39645/ using document root <none> and password file <none>
I20260807 02:26:00.635252 24686 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:00.635355 24686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:00.635592 24686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:00.637881 24686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "6d5af9e3a943492b8f78de1292e6a9e3"
format_stamp: "Formatted at 2026-08-07 02:26:00 on dist-test-slave-69rg"
server_key: "76e6e685b7d9038c1a3a1d343e240826"
server_key_iv: "611ed3142b3f644ec8a3ad48dcbb60a2"
server_key_version: "encryptionkey@0"
I20260807 02:26:00.638526 24686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "6d5af9e3a943492b8f78de1292e6a9e3"
format_stamp: "Formatted at 2026-08-07 02:26:00 on dist-test-slave-69rg"
server_key: "76e6e685b7d9038c1a3a1d343e240826"
server_key_iv: "611ed3142b3f644ec8a3ad48dcbb60a2"
server_key_version: "encryptionkey@0"
I20260807 02:26:00.642455 24686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 02:26:00.645148 24701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:00.646730 24686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:26:00.646878 24686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "6d5af9e3a943492b8f78de1292e6a9e3"
format_stamp: "Formatted at 2026-08-07 02:26:00 on dist-test-slave-69rg"
server_key: "76e6e685b7d9038c1a3a1d343e240826"
server_key_iv: "611ed3142b3f644ec8a3ad48dcbb60a2"
server_key_version: "encryptionkey@0"
I20260807 02:26:00.647003 24686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:00.663166 24686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:00.663820 24686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:00.664016 24686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:00.664628 24686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:26:00.665550 24686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:26:00.665637 24686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:00.665726 24686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:26:00.665776 24686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:00.674742 24686 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:34649
I20260807 02:26:00.674778 24818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:34649 every 8 connection(s)
I20260807 02:26:00.675701 24686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:26:00.682052 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 24686
I20260807 02:26:00.682137 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:26:00.682397 4981 external_mini_cluster.cc:1428] Setting key 5cccccaf9df329a63010371e140e220c
I20260807 02:26:00.685094 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:00.677467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:46951 (local address 127.4.221.126:37351)
0807 02:26:00.677626 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:00.677631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:00.678728 (+ 1097us) server_negotiation.cc:407] Connection header received
0807 02:26:00.680020 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:00.680022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:00.680068 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:00.680141 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:00.680959 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.681512 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.683027 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.683336 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.683440 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:00.684657 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:00.684659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:00.684661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:00.684672 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:00.684710 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:00.684721 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:00.684796 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:00.684937 (+ 141us) server_negotiation.cc:299] Negotiation successful
0807 02:26:00.684987 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 02:26:00.685950 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:00.677936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:37351 (local address 127.4.221.67:46951)
0807 02:26:00.678570 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:00.678611 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:00.679843 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:00.680294 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:00.680300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:00.680384 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:00.680826 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:00.680838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.681654 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.681657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:00.682886 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:00.682900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.683509 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.683514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:00.683712 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:00.684375 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:26:00.684377 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:00.684429 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:26:00.684507 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:00.684812 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:00.684819 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:00.684824 (+ 5us) client_negotiation.cc:769] Sending connection context
0807 02:26:00.684921 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 02:26:00.685125 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":124,"threads_started":1}
I20260807 02:26:00.687047 24819 heartbeater.cc:347] Connected to a master server at 127.4.221.126:37351
I20260807 02:26:00.687238 24819 heartbeater.cc:464] Registering TS with master...
I20260807 02:26:00.687668 24819 heartbeater.cc:511] Master 127.4.221.126:37351 requested a full tablet report, sending...
I20260807 02:26:00.688607 24350 ts_manager.cc:194] Registered new tserver with Master: 6d5af9e3a943492b8f78de1292e6a9e3 (127.4.221.67:34649)
I20260807 02:26:00.689181 24350 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.221.67:46951
I20260807 02:26:00.696347 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:26:00.702270 24390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:00.698499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46606 (local address 127.4.221.126:37351)
0807 02:26:00.698724 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:00.698727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:00.698790 (+ 63us) server_negotiation.cc:407] Connection header received
0807 02:26:00.698913 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:00.698915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:00.698952 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:00.699016 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:00.699252 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.699743 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.700568 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.700783 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.700859 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:00.701772 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:00.701775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:00.701777 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:00.701788 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:00.701806 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:00.701862 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:00.701978 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:00.702067 (+ 89us) server_negotiation.cc:299] Negotiation successful
0807 02:26:00.702119 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260807 02:26:00.705379 24350 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:26:00.708509 24350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:26:00.723196 24831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:00.717094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:43253 (local address 127.0.0.1:42956)
0807 02:26:00.717877 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:00.717894 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:00.717981 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:00.718732 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:00.718735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:00.718755 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:26:00.718852 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:00.718858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.720091 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.720095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:00.721741 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:00.721754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.722647 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.722651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:00.722756 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:00.722793 (+ 37us) client_negotiation.cc:769] Sending connection context
0807 02:26:00.722899 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 02:26:00.723037 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":212,"thread_start_us":127,"threads_started":1}
I20260807 02:26:00.723392 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:00.717606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:34649 (local address 127.0.0.1:46096)
0807 02:26:00.718389 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:00.718405 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:00.718485 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:00.719189 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:00.719194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:00.719213 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:26:00.719285 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:00.719290 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.720569 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.720572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:00.721652 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:00.721664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.722982 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.722988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:00.723066 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:00.723085 (+ 19us) client_negotiation.cc:769] Sending connection context
0807 02:26:00.723180 (+ 95us) client_negotiation.cc:240] Negotiation successful
0807 02:26:00.723249 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":67,"thread_start_us":98,"threads_started":1}
I20260807 02:26:00.723497 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:00.717098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52550 (local address 127.4.221.66:40643)
0807 02:26:00.717604 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:00.717618 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:00.717646 (+ 28us) server_negotiation.cc:407] Connection header received
0807 02:26:00.717736 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:00.717744 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:00.717980 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:00.718754 (+ 774us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:26:00.719042 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.720152 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.721854 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.722622 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.723034 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:00.723213 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:00.723261 (+ 48us) server_negotiation.cc:299] Negotiation successful
0807 02:26:00.723345 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":146,"threads_started":1}
I20260807 02:26:00.723562 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:00.716908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:40643 (local address 127.0.0.1:52550)
0807 02:26:00.717404 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:00.717452 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:00.717581 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:00.718223 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:00.718229 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:00.718257 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:26:00.718442 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:00.718453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.720267 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.720270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:00.721708 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:00.721718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.722758 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.722761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:00.722848 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:00.722872 (+ 24us) client_negotiation.cc:769] Sending connection context
0807 02:26:00.722925 (+ 53us) client_negotiation.cc:240] Negotiation successful
0807 02:26:00.723421 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":547,"thread_start_us":69,"threads_started":1}
I20260807 02:26:00.723872 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:00.718170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.4.221.67:34649)
0807 02:26:00.718619 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:00.718632 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:00.718656 (+ 24us) server_negotiation.cc:407] Connection header received
0807 02:26:00.718731 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:00.718736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:00.718908 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:00.719040 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:26:00.719419 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.720454 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.721805 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.722613 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.722733 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:00.722917 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:00.723560 (+ 643us) server_negotiation.cc:299] Negotiation successful
0807 02:26:00.723725 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":149,"threads_started":1}
I20260807 02:26:00.723248 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:00.717702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.4.221.65:43253)
0807 02:26:00.718249 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:00.718261 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:00.718287 (+ 26us) server_negotiation.cc:407] Connection header received
0807 02:26:00.718368 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:00.718373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:00.718550 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:00.719007 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:26:00.719156 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.719968 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.721981 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:00.722502 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:00.722628 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:00.722826 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:00.722939 (+ 113us) server_negotiation.cc:299] Negotiation successful
0807 02:26:00.723076 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":178,"threads_started":1}
I20260807 02:26:00.726577 24477 tablet_service.cc:1467] Processing CreateTablet for tablet 9790e96d8a5440d4a0707da5a59df576 (DEFAULT_TABLE table=test-table [id=5571f6d76bd64b61aa982fc9e2ad6640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:26:00.726603 24749 tablet_service.cc:1467] Processing CreateTablet for tablet 9790e96d8a5440d4a0707da5a59df576 (DEFAULT_TABLE table=test-table [id=5571f6d76bd64b61aa982fc9e2ad6640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:26:00.726603 24614 tablet_service.cc:1467] Processing CreateTablet for tablet 9790e96d8a5440d4a0707da5a59df576 (DEFAULT_TABLE table=test-table [id=5571f6d76bd64b61aa982fc9e2ad6640]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:26:00.727555 24477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9790e96d8a5440d4a0707da5a59df576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:00.727596 24749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9790e96d8a5440d4a0707da5a59df576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:00.727596 24614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9790e96d8a5440d4a0707da5a59df576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:00.734508 24837 tablet_bootstrap.cc:492] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc: Bootstrap starting.
I20260807 02:26:00.734874 24836 tablet_bootstrap.cc:492] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746: Bootstrap starting.
I20260807 02:26:00.736958 24836 tablet_bootstrap.cc:654] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:00.736958 24837 tablet_bootstrap.cc:654] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:00.737290 24838 tablet_bootstrap.cc:492] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3: Bootstrap starting.
I20260807 02:26:00.737928 24837 log.cc:824] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:00.737928 24836 log.cc:824] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:00.739617 24838 tablet_bootstrap.cc:654] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:00.740609 24838 log.cc:824] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:00.741312 24836 tablet_bootstrap.cc:492] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746: No bootstrap required, opened a new log
I20260807 02:26:00.741547 24836 ts_tablet_manager.cc:1397] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260807 02:26:00.742511 24837 tablet_bootstrap.cc:492] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc: No bootstrap required, opened a new log
I20260807 02:26:00.742715 24837 ts_tablet_manager.cc:1397] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260807 02:26:00.742726 24838 tablet_bootstrap.cc:492] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3: No bootstrap required, opened a new log
I20260807 02:26:00.742883 24838 ts_tablet_manager.cc:1397] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 02:26:00.745689 24836 raft_consensus.cc:348] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781d84fe860e43d0ae7ecfb1d12f44cc" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40643 } } peers { permanent_uuid: "476905c514f54b1d863aaa8cf7523746" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 43253 } } peers { permanent_uuid: "6d5af9e3a943492b8f78de1292e6a9e3" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 34649 } }
I20260807 02:26:00.745988 24836 raft_consensus.cc:374] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:00.746039 24836 raft_consensus.cc:729] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476905c514f54b1d863aaa8cf7523746, State: Initialized, Role: FOLLOWER
I20260807 02:26:00.745896 24837 raft_consensus.cc:348] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781d84fe860e43d0ae7ecfb1d12f44cc" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40643 } } peers { permanent_uuid: "476905c514f54b1d863aaa8cf7523746" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 43253 } } peers { permanent_uuid: "6d5af9e3a943492b8f78de1292e6a9e3" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 34649 } }
I20260807 02:26:00.746177 24837 raft_consensus.cc:374] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:00.746236 24837 raft_consensus.cc:729] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 781d84fe860e43d0ae7ecfb1d12f44cc, State: Initialized, Role: FOLLOWER
I20260807 02:26:00.746500 24836 consensus_queue.cc:260] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781d84fe860e43d0ae7ecfb1d12f44cc" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40643 } } peers { permanent_uuid: "476905c514f54b1d863aaa8cf7523746" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 43253 } } peers { permanent_uuid: "6d5af9e3a943492b8f78de1292e6a9e3" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 34649 } }
I20260807 02:26:00.746680 24837 consensus_queue.cc:260] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781d84fe860e43d0ae7ecfb1d12f44cc" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40643 } } peers { permanent_uuid: "476905c514f54b1d863aaa8cf7523746" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 43253 } } peers { permanent_uuid: "6d5af9e3a943492b8f78de1292e6a9e3" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 34649 } }
I20260807 02:26:00.747171 24838 raft_consensus.cc:348] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781d84fe860e43d0ae7ecfb1d12f44cc" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40643 } } peers { permanent_uuid: "476905c514f54b1d863aaa8cf7523746" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 43253 } } peers { permanent_uuid: "6d5af9e3a943492b8f78de1292e6a9e3" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 34649 } }
I20260807 02:26:00.747362 24683 heartbeater.cc:503] Master 127.4.221.126:37351 was elected leader, sending a full tablet report...
I20260807 02:26:00.747402 24838 raft_consensus.cc:374] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:00.747450 24838 raft_consensus.cc:729] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5af9e3a943492b8f78de1292e6a9e3, State: Initialized, Role: FOLLOWER
I20260807 02:26:00.747573 24836 ts_tablet_manager.cc:1428] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260807 02:26:00.747684 24837 ts_tablet_manager.cc:1428] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260807 02:26:00.747882 24546 heartbeater.cc:503] Master 127.4.221.126:37351 was elected leader, sending a full tablet report...
I20260807 02:26:00.747855 24838 consensus_queue.cc:260] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781d84fe860e43d0ae7ecfb1d12f44cc" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40643 } } peers { permanent_uuid: "476905c514f54b1d863aaa8cf7523746" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 43253 } } peers { permanent_uuid: "6d5af9e3a943492b8f78de1292e6a9e3" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 34649 } }
I20260807 02:26:00.748575 24819 heartbeater.cc:503] Master 127.4.221.126:37351 was elected leader, sending a full tablet report...
I20260807 02:26:00.749006 24838 ts_tablet_manager.cc:1428] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260807 02:26:00.787902 24547 tablet.cc:2369] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:26:00.926731 24820 tablet.cc:2369] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:26:00.990392 24684 tablet.cc:2369] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:26:01.098093 24844 raft_consensus.cc:482] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:26:01.098369 24844 raft_consensus.cc:504] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781d84fe860e43d0ae7ecfb1d12f44cc" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40643 } } peers { permanent_uuid: "476905c514f54b1d863aaa8cf7523746" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 43253 } } peers { permanent_uuid: "6d5af9e3a943492b8f78de1292e6a9e3" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 34649 } }
I20260807 02:26:01.099532 24844 leader_election.cc:288] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 476905c514f54b1d863aaa8cf7523746 (127.4.221.65:43253), 6d5af9e3a943492b8f78de1292e6a9e3 (127.4.221.67:34649)
I20260807 02:26:01.103747 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:01.100025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:54833 (local address 127.4.221.65:43253)
0807 02:26:01.100233 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:01.100240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:01.100328 (+ 88us) server_negotiation.cc:407] Connection header received
0807 02:26:01.100549 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:01.100553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:01.100625 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:01.100716 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:26:01.101098 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:01.101624 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:01.102766 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:01.103175 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:01.103311 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:01.103362 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:01.103546 (+ 184us) server_negotiation.cc:299] Negotiation successful
0807 02:26:01.103617 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260807 02:26:01.103765 24846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:01.099866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:43253 (local address 127.4.221.66:54833)
0807 02:26:01.100213 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:01.100231 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:01.100389 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:01.100757 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:01.100761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:01.100788 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:26:01.100958 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:01.100967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:01.101748 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:01.101754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:01.102619 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:01.102629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:01.103306 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:01.103310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:01.103410 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:01.103438 (+ 28us) client_negotiation.cc:769] Sending connection context
0807 02:26:01.103527 (+ 89us) client_negotiation.cc:240] Negotiation successful
0807 02:26:01.103602 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"spinlock_wait_cycles":28416,"thread_start_us":133,"threads_started":1}
I20260807 02:26:01.104329 24847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:01.100371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:34649 (local address 127.4.221.66:60393)
0807 02:26:01.100744 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:01.100758 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:01.100887 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:01.101228 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:01.101232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:01.101252 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:26:01.101397 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:01.101404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:01.102226 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:01.102232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:01.103087 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:01.103097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:01.103843 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:01.103846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:01.103972 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:01.104007 (+ 35us) client_negotiation.cc:769] Sending connection context
0807 02:26:01.104089 (+ 82us) client_negotiation.cc:240] Negotiation successful
0807 02:26:01.104167 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"spinlock_wait_cycles":3200,"thread_start_us":112,"threads_started":1}
I20260807 02:26:01.104403 24497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9790e96d8a5440d4a0707da5a59df576" candidate_uuid: "781d84fe860e43d0ae7ecfb1d12f44cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476905c514f54b1d863aaa8cf7523746" is_pre_election: true
I20260807 02:26:01.104768 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:01.100476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:60393 (local address 127.4.221.67:34649)
0807 02:26:01.100600 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:01.100605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:01.100823 (+ 218us) server_negotiation.cc:407] Connection header received
0807 02:26:01.101014 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:01.101016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:01.101060 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:01.101215 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:26:01.101528 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:01.102094 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:01.103232 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:01.103689 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:01.103843 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:01.103891 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:01.104547 (+ 656us) server_negotiation.cc:299] Negotiation successful
0807 02:26:01.104644 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 02:26:01.104759 24497 raft_consensus.cc:2454] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 781d84fe860e43d0ae7ecfb1d12f44cc in term 0.
I20260807 02:26:01.105350 24565 leader_election.cc:302] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476905c514f54b1d863aaa8cf7523746, 781d84fe860e43d0ae7ecfb1d12f44cc; no voters:
I20260807 02:26:01.105506 24769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9790e96d8a5440d4a0707da5a59df576" candidate_uuid: "781d84fe860e43d0ae7ecfb1d12f44cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5af9e3a943492b8f78de1292e6a9e3" is_pre_election: true
I20260807 02:26:01.105682 24844 raft_consensus.cc:2781] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:26:01.105810 24844 raft_consensus.cc:482] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:26:01.105898 24844 raft_consensus.cc:3037] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:01.105804 24769 raft_consensus.cc:2454] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 781d84fe860e43d0ae7ecfb1d12f44cc in term 0.
I20260807 02:26:01.107049 24844 raft_consensus.cc:504] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781d84fe860e43d0ae7ecfb1d12f44cc" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40643 } } peers { permanent_uuid: "476905c514f54b1d863aaa8cf7523746" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 43253 } } peers { permanent_uuid: "6d5af9e3a943492b8f78de1292e6a9e3" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 34649 } }
I20260807 02:26:01.107409 24844 leader_election.cc:288] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [CANDIDATE]: Term 1 election: Requested vote from peers 476905c514f54b1d863aaa8cf7523746 (127.4.221.65:43253), 6d5af9e3a943492b8f78de1292e6a9e3 (127.4.221.67:34649)
I20260807 02:26:01.107963 24497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9790e96d8a5440d4a0707da5a59df576" candidate_uuid: "781d84fe860e43d0ae7ecfb1d12f44cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476905c514f54b1d863aaa8cf7523746"
I20260807 02:26:01.108004 24769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9790e96d8a5440d4a0707da5a59df576" candidate_uuid: "781d84fe860e43d0ae7ecfb1d12f44cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5af9e3a943492b8f78de1292e6a9e3"
I20260807 02:26:01.108078 24497 raft_consensus.cc:3037] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:01.108194 24769 raft_consensus.cc:3037] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:01.109185 24769 raft_consensus.cc:2454] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 781d84fe860e43d0ae7ecfb1d12f44cc in term 1.
I20260807 02:26:01.109202 24497 raft_consensus.cc:2454] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 781d84fe860e43d0ae7ecfb1d12f44cc in term 1.
I20260807 02:26:01.109510 24565 leader_election.cc:302] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d5af9e3a943492b8f78de1292e6a9e3, 781d84fe860e43d0ae7ecfb1d12f44cc; no voters:
I20260807 02:26:01.109707 24844 raft_consensus.cc:2781] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:01.109978 24844 raft_consensus.cc:686] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [term 1 LEADER]: Becoming Leader. State: Replica: 781d84fe860e43d0ae7ecfb1d12f44cc, State: Running, Role: LEADER
I20260807 02:26:01.110325 24844 consensus_queue.cc:237] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781d84fe860e43d0ae7ecfb1d12f44cc" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40643 } } peers { permanent_uuid: "476905c514f54b1d863aaa8cf7523746" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 43253 } } peers { permanent_uuid: "6d5af9e3a943492b8f78de1292e6a9e3" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 34649 } }
I20260807 02:26:01.113441 24348 catalog_manager.cc:5495] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 781d84fe860e43d0ae7ecfb1d12f44cc (127.4.221.66). New cstate: current_term: 1 leader_uuid: "781d84fe860e43d0ae7ecfb1d12f44cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "781d84fe860e43d0ae7ecfb1d12f44cc" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 40643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476905c514f54b1d863aaa8cf7523746" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 43253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d5af9e3a943492b8f78de1292e6a9e3" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 34649 } health_report { overall_health: UNKNOWN } } }
I20260807 02:26:01.186153 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:01.181993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.4.221.66:40643)
0807 02:26:01.182159 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:01.182165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:01.182182 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:26:01.182278 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:01.182280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:01.182353 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:01.182468 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:26:01.182740 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:01.183338 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:01.184318 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:01.184584 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:01.184677 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:01.185067 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:26:01.185611 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:26:01.185740 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:01.185877 (+ 137us) server_negotiation.cc:299] Negotiation successful
0807 02:26:01.185941 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 02:26:01.194130 24769 raft_consensus.cc:1261] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3 [term 1 FOLLOWER]: Refusing update from remote peer 781d84fe860e43d0ae7ecfb1d12f44cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:26:01.194130 24497 raft_consensus.cc:1261] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746 [term 1 FOLLOWER]: Refusing update from remote peer 781d84fe860e43d0ae7ecfb1d12f44cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:26:01.195070 24844 consensus_queue.cc:1035] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "476905c514f54b1d863aaa8cf7523746" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 43253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:26:01.195340 24848 consensus_queue.cc:1035] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d5af9e3a943492b8f78de1292e6a9e3" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:26:01.204295 24851 mvcc.cc:205] Tried to move back new op lower bound from 7315740922640941056 to 7315740922310389760. Current Snapshot: MvccSnapshot[applied={T|T < 7315740922640941056}]
I20260807 02:26:01.207239 24854 mvcc.cc:205] Tried to move back new op lower bound from 7315740922640941056 to 7315740922310389760. Current Snapshot: MvccSnapshot[applied={T|T < 7315740922640941056}]
I20260807 02:26:01.216136 24349 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:26:01.216348 24349 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:26:01.219574 24349 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70aa3c17017e45fa8afcafb64e5befbd: Sending DeleteTablet for 3 replicas of tablet 9790e96d8a5440d4a0707da5a59df576
I20260807 02:26:01.220680 24614 tablet_service.cc:1514] Processing DeleteTablet for tablet 9790e96d8a5440d4a0707da5a59df576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:26:01 UTC) from {username='slave'} at 127.0.0.1:52550
I20260807 02:26:01.220757 24749 tablet_service.cc:1514] Processing DeleteTablet for tablet 9790e96d8a5440d4a0707da5a59df576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:26:01 UTC) from {username='slave'} at 127.0.0.1:46096
I20260807 02:26:01.220947 24477 tablet_service.cc:1514] Processing DeleteTablet for tablet 9790e96d8a5440d4a0707da5a59df576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:26:01 UTC) from {username='slave'} at 127.0.0.1:42956
I20260807 02:26:01.221499 24863 tablet_replica.cc:330] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746: stopping tablet replica
I20260807 02:26:01.221503 24861 tablet_replica.cc:330] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc: stopping tablet replica
I20260807 02:26:01.221939 24863 raft_consensus.cc:2229] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:26:01.221939 24861 raft_consensus.cc:2229] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:26:01.222389 24861 raft_consensus.cc:2258] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:26:01.222434 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 24412
I20260807 02:26:01.222690 24863 raft_consensus.cc:2258] T 9790e96d8a5440d4a0707da5a59df576 P 476905c514f54b1d863aaa8cf7523746 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:26:01.224671 24862 tablet_replica.cc:330] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3: stopping tablet replica
I20260807 02:26:01.225062 24862 raft_consensus.cc:2229] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:26:01.225399 24862 raft_consensus.cc:2258] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:26:01.225740 24861 ts_tablet_manager.cc:1905] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:26:01.227332 24862 ts_tablet_manager.cc:1905] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:26:01.229394 24861 ts_tablet_manager.cc:1918] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:26:01.229552 24861 log.cc:1196] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/9790e96d8a5440d4a0707da5a59df576
I20260807 02:26:01.229931 24861 ts_tablet_manager.cc:1939] T 9790e96d8a5440d4a0707da5a59df576 P 781d84fe860e43d0ae7ecfb1d12f44cc: Deleting consensus metadata
I20260807 02:26:01.230623 24862 ts_tablet_manager.cc:1918] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:26:01.230731 24862 log.cc:1196] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/9790e96d8a5440d4a0707da5a59df576
I20260807 02:26:01.231076 24862 ts_tablet_manager.cc:1939] T 9790e96d8a5440d4a0707da5a59df576 P 6d5af9e3a943492b8f78de1292e6a9e3: Deleting consensus metadata
I20260807 02:26:01.231432 24336 catalog_manager.cc:4841] TS 781d84fe860e43d0ae7ecfb1d12f44cc (127.4.221.66:40643): tablet 9790e96d8a5440d4a0707da5a59df576 (table test-table [id=5571f6d76bd64b61aa982fc9e2ad6640]) successfully deleted
I20260807 02:26:01.232347 24333 catalog_manager.cc:4841] TS 6d5af9e3a943492b8f78de1292e6a9e3 (127.4.221.67:34649): tablet 9790e96d8a5440d4a0707da5a59df576 (table test-table [id=5571f6d76bd64b61aa982fc9e2ad6640]) successfully deleted
W20260807 02:26:01.238379 24333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:43253 (error 108)
I20260807 02:26:01.238587 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 24549
I20260807 02:26:01.241868 24865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:01.241353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:43253 (local address 127.0.0.1:42966)
0807 02:26:01.241617 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:01.241704 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:43253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
W20260807 02:26:01.242066 24333 catalog_manager.cc:4584] TS 476905c514f54b1d863aaa8cf7523746 (127.4.221.65:43253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9790e96d8a5440d4a0707da5a59df576: Network error: Client connection negotiation failed: client connection to 127.4.221.65:43253: connect: Connection refused (error 111)
I20260807 02:26:01.246908 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 24686
I20260807 02:26:01.254570 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 24316
2026-08-07T02:26:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1418 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:01 dist-test-slave-69rg krb5kdc[24869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:01 dist-test-slave-69rg krb5kdc[24869](info): set up 2 sockets
Aug 07 02:26:01 dist-test-slave-69rg krb5kdc[24869](info): commencing operation
krb5kdc: starting...
W20260807 02:26:02.349114 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:02 dist-test-slave-69rg krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:26:02.485127 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:40099
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:40099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:02.596642 24883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:02.596962 24883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:02.597039 24883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:02.601104 24883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:02.601197 24883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:02.601234 24883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:02.601295 24883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:02.601349 24883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:02.606544 24883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:40099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:40099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:02.607559 24883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:02.608554 24883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:02.616891 24892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:02.616930 24889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:02.616917 24890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:02.617317 24883 server_base.cc:1034] running on GCE node
I20260807 02:26:02.617913 24883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:02.618976 24883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:02.620172 24883 hybrid_clock.cc:648] HybridClock initialized: now 1786069562620157 us; error 35 us; skew 500 ppm
Aug 07 02:26:02 dist-test-slave-69rg krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069562, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:02.623064 24883 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:26:02.623929 24883 webserver.cc:460] Webserver started at http://127.4.221.126:34399/ using document root <none> and password file <none>
I20260807 02:26:02.624485 24883 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:02.624579 24883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:02.624823 24883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:02.626912 24883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "f3aa6ae59c5346a6ba728d44268a8668"
format_stamp: "Formatted at 2026-08-07 02:26:02 on dist-test-slave-69rg"
server_key: "71eac5e634dcbcb2b57da53d4a085405"
server_key_iv: "1f9462919785fab366ffedd45e04f1a3"
server_key_version: "encryptionkey@0"
I20260807 02:26:02.627487 24883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "f3aa6ae59c5346a6ba728d44268a8668"
format_stamp: "Formatted at 2026-08-07 02:26:02 on dist-test-slave-69rg"
server_key: "71eac5e634dcbcb2b57da53d4a085405"
server_key_iv: "1f9462919785fab366ffedd45e04f1a3"
server_key_version: "encryptionkey@0"
I20260807 02:26:02.631106 24883 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:26:02.633481 24899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:02.634819 24883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:26:02.634974 24883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "f3aa6ae59c5346a6ba728d44268a8668"
format_stamp: "Formatted at 2026-08-07 02:26:02 on dist-test-slave-69rg"
server_key: "71eac5e634dcbcb2b57da53d4a085405"
server_key_iv: "1f9462919785fab366ffedd45e04f1a3"
server_key_version: "encryptionkey@0"
I20260807 02:26:02.635089 24883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:02.653183 24883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:02.656441 24883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:02.656656 24883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:02.663941 24883 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:40099
I20260807 02:26:02.663967 24954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:40099 every 8 connection(s)
I20260807 02:26:02.664909 24883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:02.667680 24955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:02.672248 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 24883
I20260807 02:26:02.672407 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:02.672777 4981 external_mini_cluster.cc:1428] Setting key 5bc0efcc1ef696989f578f1760227e2f
I20260807 02:26:02.673967 24955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668: Bootstrap starting.
I20260807 02:26:02.676275 24955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:02.677035 24955 log.cc:824] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:02.678983 24955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668: No bootstrap required, opened a new log
Aug 07 02:26:02 dist-test-slave-69rg krb5kdc[24869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:26:02.683060 24955 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3aa6ae59c5346a6ba728d44268a8668" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40099 } }
I20260807 02:26:02.683283 24955 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:02.683318 24955 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3aa6ae59c5346a6ba728d44268a8668, State: Initialized, Role: FOLLOWER
I20260807 02:26:02.683823 24955 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3aa6ae59c5346a6ba728d44268a8668" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40099 } }
I20260807 02:26:02.683975 24955 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:02.684072 24955 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:02.684202 24955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:02.685282 24955 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3aa6ae59c5346a6ba728d44268a8668" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40099 } }
I20260807 02:26:02.685670 24955 leader_election.cc:302] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3aa6ae59c5346a6ba728d44268a8668; no voters:
I20260807 02:26:02.686003 24955 leader_election.cc:288] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:02.686113 24960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:02.686373 24960 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [term 1 LEADER]: Becoming Leader. State: Replica: f3aa6ae59c5346a6ba728d44268a8668, State: Running, Role: LEADER
I20260807 02:26:02.686678 24960 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3aa6ae59c5346a6ba728d44268a8668" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40099 } }
I20260807 02:26:02.687362 24955 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:02.688023 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:02.674277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.4.221.126:40099)
0807 02:26:02.674722 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:02.674738 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:02.674772 (+ 34us) server_negotiation.cc:407] Connection header received
0807 02:26:02.675400 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:02.675417 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:02.675743 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:02.676072 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:02.676924 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:02.678037 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:02.678953 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:02.679488 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:02.679672 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:02.682587 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:02.682593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:02.682604 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:02.682630 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:02.684690 (+ 2060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:02.685171 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:02.685175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:02.685179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:02.685285 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:02.685579 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:02.685581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:02.685582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:02.686121 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:02.686332 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:02.686734 (+ 402us) server_negotiation.cc:299] Negotiation successful
0807 02:26:02.687042 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":126,"threads_started":1}
I20260807 02:26:02.688796 24963 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3aa6ae59c5346a6ba728d44268a8668. Latest consensus state: current_term: 1 leader_uuid: "f3aa6ae59c5346a6ba728d44268a8668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3aa6ae59c5346a6ba728d44268a8668" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40099 } } }
I20260807 02:26:02.688915 24963 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:02.689210 24962 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3aa6ae59c5346a6ba728d44268a8668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3aa6ae59c5346a6ba728d44268a8668" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 40099 } } }
I20260807 02:26:02.689303 24962 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:02.690271 24972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:02.692945 24972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:02.698554 24972 catalog_manager.cc:1347] Generated new cluster ID: 3491c054ce88411382bb74646d98dc03
I20260807 02:26:02.698621 24972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:02.707867 24972 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:02.709025 24972 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:02.721787 24972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668: Generated new TSK 0
I20260807 02:26:02.722621 24972 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.221.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
Entry for principal kudu/127.4.221.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab.
I20260807 02:26:02.763007 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40099
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:02.875770 24982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:02.876086 24982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:02.876165 24982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:02.880381 24982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:02.880461 24982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:02.880591 24982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:26:02.886693 24982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.65.keytab
--principal=kudu/127.4.221.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:02.887858 24982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:02.888996 24982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:02.898313 24990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:02.898373 24987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:02.898373 24988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:02.898612 24982 server_base.cc:1034] running on GCE node
I20260807 02:26:02.898977 24982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:02.899616 24982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:02.900806 24982 hybrid_clock.cc:648] HybridClock initialized: now 1786069562900778 us; error 40 us; skew 500 ppm
Aug 07 02:26:02 dist-test-slave-69rg krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069562, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:02.903590 24982 init.cc:377] Logged in from keytab as kudu/127.4.221.65@KRBTEST.COM (short username kudu)
I20260807 02:26:02.904397 24982 webserver.cc:460] Webserver started at http://127.4.221.65:35079/ using document root <none> and password file <none>
I20260807 02:26:02.904879 24982 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:02.904937 24982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:02.905104 24982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:02.907433 24982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "9867ca656c6a42ee982a35324f0f14d7"
format_stamp: "Formatted at 2026-08-07 02:26:02 on dist-test-slave-69rg"
server_key: "b6916de94b1f783ee5b1521316b70d6d"
server_key_iv: "113f2aced6a835444016dce579e91676"
server_key_version: "encryptionkey@0"
I20260807 02:26:02.907930 24982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "9867ca656c6a42ee982a35324f0f14d7"
format_stamp: "Formatted at 2026-08-07 02:26:02 on dist-test-slave-69rg"
server_key: "b6916de94b1f783ee5b1521316b70d6d"
server_key_iv: "113f2aced6a835444016dce579e91676"
server_key_version: "encryptionkey@0"
I20260807 02:26:02.911520 24982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:26:02.913954 24997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:02.915236 24982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 02:26:02.915370 24982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "9867ca656c6a42ee982a35324f0f14d7"
format_stamp: "Formatted at 2026-08-07 02:26:02 on dist-test-slave-69rg"
server_key: "b6916de94b1f783ee5b1521316b70d6d"
server_key_iv: "113f2aced6a835444016dce579e91676"
server_key_version: "encryptionkey@0"
I20260807 02:26:02.915482 24982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:02.933091 24982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:02.936383 24982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:02.936600 24982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:02.937227 24982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:26:02.938163 24982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:26:02.938213 24982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:02.938271 24982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:26:02.938302 24982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:02.947701 24982 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:34001
I20260807 02:26:02.947798 25114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:34001 every 8 connection(s)
I20260807 02:26:02.948609 24982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:26:02.950464 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 24982
I20260807 02:26:02.950600 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:26:02.950922 4981 external_mini_cluster.cc:1428] Setting key 9cbb47c361355214cf9b78393c9d2747
Aug 07 02:26:02 dist-test-slave-69rg krb5kdc[24869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069562, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.65@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.221.66@KRBTEST.COM; defaulting to no policy
I20260807 02:26:02.962869 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:02.950369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:51951 (local address 127.4.221.126:40099)
0807 02:26:02.950549 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:02.950555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:02.951496 (+ 941us) server_negotiation.cc:407] Connection header received
0807 02:26:02.952564 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:02.952568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:02.952626 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:02.952725 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:02.953875 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:02.954650 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:02.956171 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:02.956434 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:02.956549 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:02.959024 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:02.959029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:02.959032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:02.959062 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:02.960769 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:02.961260 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:02.961263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:02.961265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:02.961350 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:02.961806 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:02.961809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:02.961811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:02.962018 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:02.962098 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:02.962644 (+ 546us) server_negotiation.cc:299] Negotiation successful
0807 02:26:02.962746 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 02:26:02.963801 25118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:02.950795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40099 (local address 127.4.221.65:51951)
0807 02:26:02.951316 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:02.951383 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:02.952368 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:02.952870 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:02.952877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:02.953253 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:02.953707 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:02.953719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:02.954785 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:02.954791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:02.956039 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:02.956049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:02.956589 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:02.956594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:02.956756 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:02.957347 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:02.957363 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:02.958856 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:02.960899 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:02.960904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:02.960914 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:02.961160 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:02.961480 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:02.961483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:02.961485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:02.961698 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:02.962128 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:02.962134 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:02.962362 (+ 228us) client_negotiation.cc:769] Sending connection context
0807 02:26:02.962532 (+ 170us) client_negotiation.cc:240] Negotiation successful
0807 02:26:02.962849 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":140,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.66@KRBTEST.COM" created.
I20260807 02:26:02.965168 25115 heartbeater.cc:347] Connected to a master server at 127.4.221.126:40099
I20260807 02:26:02.965410 25115 heartbeater.cc:464] Registering TS with master...
I20260807 02:26:02.965943 25115 heartbeater.cc:511] Master 127.4.221.126:40099 requested a full tablet report, sending...
I20260807 02:26:02.967590 24917 ts_manager.cc:194] Registered new tserver with Master: 9867ca656c6a42ee982a35324f0f14d7 (127.4.221.65:34001)
I20260807 02:26:02.969247 24917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.65@KRBTEST.COM'} at 127.4.221.65:51951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
Entry for principal kudu/127.4.221.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab.
I20260807 02:26:02.981177 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40099
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:03.092486 25121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:03.092756 25121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:03.092831 25121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:03.096871 25121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:03.096961 25121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:03.097051 25121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:26:03.102316 25121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.66.keytab
--principal=kudu/127.4.221.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:03.103338 25121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:03.104308 25121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:03.113473 25126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:03.113508 25127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:03.113592 25121 server_base.cc:1034] running on GCE node
W20260807 02:26:03.113508 25130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:03.114092 25121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:03.114748 25121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:03.115942 25121 hybrid_clock.cc:648] HybridClock initialized: now 1786069563115923 us; error 28 us; skew 500 ppm
Aug 07 02:26:03 dist-test-slave-69rg krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069563, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:03.118829 25121 init.cc:377] Logged in from keytab as kudu/127.4.221.66@KRBTEST.COM (short username kudu)
I20260807 02:26:03.119779 25121 webserver.cc:460] Webserver started at http://127.4.221.66:41405/ using document root <none> and password file <none>
I20260807 02:26:03.120321 25121 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:03.120410 25121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:03.120674 25121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:03.122720 25121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "977a9133c2a54c77855b31e7b7f4311f"
format_stamp: "Formatted at 2026-08-07 02:26:03 on dist-test-slave-69rg"
server_key: "a353161ebbbc840696688155524af4eb"
server_key_iv: "8e26edeb21d70951bd7804bfa53bf962"
server_key_version: "encryptionkey@0"
I20260807 02:26:03.123277 25121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "977a9133c2a54c77855b31e7b7f4311f"
format_stamp: "Formatted at 2026-08-07 02:26:03 on dist-test-slave-69rg"
server_key: "a353161ebbbc840696688155524af4eb"
server_key_iv: "8e26edeb21d70951bd7804bfa53bf962"
server_key_version: "encryptionkey@0"
I20260807 02:26:03.126827 25121 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260807 02:26:03.129112 25137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:03.130338 25121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 02:26:03.130470 25121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "977a9133c2a54c77855b31e7b7f4311f"
format_stamp: "Formatted at 2026-08-07 02:26:03 on dist-test-slave-69rg"
server_key: "a353161ebbbc840696688155524af4eb"
server_key_iv: "8e26edeb21d70951bd7804bfa53bf962"
server_key_version: "encryptionkey@0"
I20260807 02:26:03.130578 25121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:03.147861 25121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:03.150739 25121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:03.150934 25121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:03.151484 25121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:26:03.152259 25121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:26:03.152331 25121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:03.152405 25121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:26:03.152448 25121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:03.161145 25121 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:43307
I20260807 02:26:03.161187 25255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:43307 every 8 connection(s)
I20260807 02:26:03.162035 25121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:26:03.168020 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 25121
I20260807 02:26:03.168207 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:26:03.168548 4981 external_mini_cluster.cc:1428] Setting key 89793c349196ae2cbc42ab7f7860dec1
Aug 07 02:26:03 dist-test-slave-69rg krb5kdc[24869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069563, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.66@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:26:03.175897 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.163722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:48933 (local address 127.4.221.126:40099)
0807 02:26:03.163864 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:03.163868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:03.164664 (+ 796us) server_negotiation.cc:407] Connection header received
0807 02:26:03.165484 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:03.165487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:03.165533 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:03.165606 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:03.166704 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.167239 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.168728 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.168989 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.169118 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.171675 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:03.171678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:03.171680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:03.171705 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:03.173443 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:03.174113 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:03.174116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:03.174118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:03.174204 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:03.174632 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:03.174635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:03.174636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:03.174817 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:03.174893 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:03.175666 (+ 773us) server_negotiation.cc:299] Negotiation successful
0807 02:26:03.175778 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 02:26:03.177057 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.164046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40099 (local address 127.4.221.66:48933)
0807 02:26:03.164487 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:03.164528 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:03.165291 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:03.165770 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:03.165776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:03.166123 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:03.166555 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:03.166567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.167344 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.167347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:03.168593 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:03.168605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.169216 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.169222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:03.169415 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.170018 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:03.170034 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:03.171495 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:03.173580 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:03.173589 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:03.173606 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:03.174015 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:03.174301 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:03.174307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:03.174311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:03.174506 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:03.174926 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:03.174935 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:03.175320 (+ 385us) client_negotiation.cc:769] Sending connection context
0807 02:26:03.175543 (+ 223us) client_negotiation.cc:240] Negotiation successful
0807 02:26:03.175909 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":108,"threads_started":1}
I20260807 02:26:03.178510 25256 heartbeater.cc:347] Connected to a master server at 127.4.221.126:40099
I20260807 02:26:03.178768 25256 heartbeater.cc:464] Registering TS with master...
I20260807 02:26:03.179354 25256 heartbeater.cc:511] Master 127.4.221.126:40099 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.221.67@KRBTEST.COM; defaulting to no policy
I20260807 02:26:03.180507 24917 ts_manager.cc:194] Registered new tserver with Master: 977a9133c2a54c77855b31e7b7f4311f (127.4.221.66:43307)
I20260807 02:26:03.181173 24917 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.66@KRBTEST.COM'} at 127.4.221.66:48933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
Entry for principal kudu/127.4.221.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab.
I20260807 02:26:03.200327 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40099
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:03.310868 25261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:03.311111 25261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:03.311157 25261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:03.315161 25261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:03.315217 25261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:03.315274 25261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:26:03.320624 25261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.67.keytab
--principal=kudu/127.4.221.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:40099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:03.321610 25261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:03.322634 25261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:03.331873 25269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:03.331908 25266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:03.331883 25267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:03.332118 25261 server_base.cc:1034] running on GCE node
I20260807 02:26:03.332593 25261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:03.333252 25261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:03.334393 25261 hybrid_clock.cc:648] HybridClock initialized: now 1786069563334369 us; error 38 us; skew 500 ppm
Aug 07 02:26:03 dist-test-slave-69rg krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069563, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:03.337177 25261 init.cc:377] Logged in from keytab as kudu/127.4.221.67@KRBTEST.COM (short username kudu)
I20260807 02:26:03.338042 25261 webserver.cc:460] Webserver started at http://127.4.221.67:43085/ using document root <none> and password file <none>
I20260807 02:26:03.338634 25261 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:03.338696 25261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:03.338855 25261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:03.340901 25261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "0eee1d4dab1c4e18a667876cb422c1bc"
format_stamp: "Formatted at 2026-08-07 02:26:03 on dist-test-slave-69rg"
server_key: "6e2507472d09d39aec0c695c52b0d5a1"
server_key_iv: "e4916c388a92f06be77d999f498970ce"
server_key_version: "encryptionkey@0"
I20260807 02:26:03.341401 25261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "0eee1d4dab1c4e18a667876cb422c1bc"
format_stamp: "Formatted at 2026-08-07 02:26:03 on dist-test-slave-69rg"
server_key: "6e2507472d09d39aec0c695c52b0d5a1"
server_key_iv: "e4916c388a92f06be77d999f498970ce"
server_key_version: "encryptionkey@0"
I20260807 02:26:03.344990 25261 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260807 02:26:03.347374 25277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:03.348644 25261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 02:26:03.348776 25261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "0eee1d4dab1c4e18a667876cb422c1bc"
format_stamp: "Formatted at 2026-08-07 02:26:03 on dist-test-slave-69rg"
server_key: "6e2507472d09d39aec0c695c52b0d5a1"
server_key_iv: "e4916c388a92f06be77d999f498970ce"
server_key_version: "encryptionkey@0"
I20260807 02:26:03.348887 25261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:03.361537 25261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:03.364218 25261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:03.364414 25261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:03.364977 25261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:26:03.365765 25261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:26:03.365876 25261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:03.365963 25261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:26:03.366015 25261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:03.374773 25261 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:45225
I20260807 02:26:03.374799 25395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:45225 every 8 connection(s)
I20260807 02:26:03.375625 25261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:26:03.377135 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 25261
I20260807 02:26:03.377238 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:26:03.377606 4981 external_mini_cluster.cc:1428] Setting key 440f2d6d0723f9b0c6264376789aff8b
Aug 07 02:26:03 dist-test-slave-69rg krb5kdc[24869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069563, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.67@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:26:03.389210 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.377464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:40429 (local address 127.4.221.126:40099)
0807 02:26:03.377629 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:03.377634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:03.378980 (+ 1346us) server_negotiation.cc:407] Connection header received
0807 02:26:03.380035 (+ 1055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:03.380038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:03.380083 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:03.380155 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:03.381276 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.381799 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.383007 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.383245 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.383323 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.385809 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:03.385812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:03.385866 (+ 54us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:03.385898 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:03.387258 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:03.387743 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:03.387746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:03.387747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:03.387891 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:03.388250 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:03.388253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:03.388254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:03.388432 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:03.388508 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:03.388948 (+ 440us) server_negotiation.cc:299] Negotiation successful
0807 02:26:03.389046 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 02:26:03.390025 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.377903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:40099 (local address 127.4.221.67:40429)
0807 02:26:03.378837 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:03.378894 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:03.379812 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:03.380338 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:03.380344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:03.380678 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:03.381137 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:03.381150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.382006 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.382009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:03.382901 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:03.382912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.383374 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.383377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:03.383513 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.384146 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:03.384162 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:03.385658 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:03.387374 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:03.387378 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:03.387388 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:03.387643 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:03.387981 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:03.387983 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:03.387985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:03.388176 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:03.388521 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:03.388526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:03.388751 (+ 225us) client_negotiation.cc:769] Sending connection context
0807 02:26:03.388900 (+ 149us) client_negotiation.cc:240] Negotiation successful
0807 02:26:03.389135 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":124,"threads_started":1}
I20260807 02:26:03.391211 25396 heartbeater.cc:347] Connected to a master server at 127.4.221.126:40099
I20260807 02:26:03.391449 25396 heartbeater.cc:464] Registering TS with master...
I20260807 02:26:03.391947 25396 heartbeater.cc:511] Master 127.4.221.126:40099 requested a full tablet report, sending...
I20260807 02:26:03.392911 24916 ts_manager.cc:194] Registered new tserver with Master: 0eee1d4dab1c4e18a667876cb422c1bc (127.4.221.67:45225)
I20260807 02:26:03.393481 24916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.221.67@KRBTEST.COM'} at 127.4.221.67:40429
I20260807 02:26:03.403398 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:26:03.412250 24958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.405521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48266 (local address 127.4.221.126:40099)
0807 02:26:03.405649 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:03.405663 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:03.405790 (+ 127us) server_negotiation.cc:407] Connection header received
0807 02:26:03.405952 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:03.405954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:03.405988 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:03.406050 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:03.406699 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.407185 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.407959 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.408163 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.408243 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.409483 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:03.409486 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:03.409488 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:03.409507 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:03.410890 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:03.411263 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:03.411266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:03.411267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:03.411342 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:03.411573 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:03.411575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:03.411576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:03.411793 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:03.411896 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:03.412050 (+ 154us) server_negotiation.cc:299] Negotiation successful
0807 02:26:03.412138 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:26:03.415627 24916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:26:03.418732 24916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:26:03.433445 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.427171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:34001 (local address 127.0.0.1:35900)
0807 02:26:03.427870 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:03.427886 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:03.428016 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:03.428823 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:03.428828 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:03.428849 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:26:03.429006 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:03.429014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.430789 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.430795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:03.431907 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:03.431917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.432917 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.432920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:03.433022 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.433051 (+ 29us) client_negotiation.cc:769] Sending connection context
0807 02:26:03.433129 (+ 78us) client_negotiation.cc:240] Negotiation successful
0807 02:26:03.433252 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
I20260807 02:26:03.433719 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.427041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:45225 (local address 127.0.0.1:42350)
0807 02:26:03.427836 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:03.427885 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:03.428343 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:03.429013 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:03.429016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:03.429031 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:26:03.429159 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:03.429170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.430594 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.430598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:03.431756 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:03.431765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.432849 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.432853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:03.432972 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.433018 (+ 46us) client_negotiation.cc:769] Sending connection context
0807 02:26:03.433118 (+ 100us) client_negotiation.cc:240] Negotiation successful
0807 02:26:03.433565 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":339,"thread_start_us":138,"threads_started":1}
I20260807 02:26:03.433852 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.427161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42350 (local address 127.4.221.67:45225)
0807 02:26:03.428607 (+ 1446us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:03.428619 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:03.428639 (+ 20us) server_negotiation.cc:407] Connection header received
0807 02:26:03.428709 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:03.428714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:03.428880 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:03.429027 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:26:03.429321 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.430444 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.431882 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.432715 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.432845 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.433006 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:03.433247 (+ 241us) server_negotiation.cc:299] Negotiation successful
0807 02:26:03.433353 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1343,"thread_start_us":166,"threads_started":1}
I20260807 02:26:03.434055 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.428106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:43307 (local address 127.0.0.1:41096)
0807 02:26:03.428355 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:03.428370 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:03.428470 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:03.429767 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:03.429771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:03.429789 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:26:03.429888 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:03.429894 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.431328 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.431332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:03.432279 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:03.432287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.433149 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.433152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:03.433244 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.433263 (+ 19us) client_negotiation.cc:769] Sending connection context
0807 02:26:03.433342 (+ 79us) client_negotiation.cc:240] Negotiation successful
0807 02:26:03.433397 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":108,"threads_started":1}
I20260807 02:26:03.434497 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.427345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35900 (local address 127.4.221.65:34001)
0807 02:26:03.427752 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:03.427769 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:03.428160 (+ 391us) server_negotiation.cc:407] Connection header received
0807 02:26:03.428253 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:03.428258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:03.428467 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:03.428610 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:26:03.429125 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.430295 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.432028 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.432786 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.434038 (+ 1252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.434193 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:03.434249 (+ 56us) server_negotiation.cc:299] Negotiation successful
0807 02:26:03.434342 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":98,"threads_started":1}
I20260807 02:26:03.434738 25413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.428285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41096 (local address 127.4.221.66:43307)
0807 02:26:03.429331 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:03.429343 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:03.429366 (+ 23us) server_negotiation.cc:407] Connection header received
0807 02:26:03.429438 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:03.429443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:03.429610 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:03.429765 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:26:03.430090 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.431184 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.432389 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.433030 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.433143 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.433276 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:03.433368 (+ 92us) server_negotiation.cc:299] Negotiation successful
0807 02:26:03.433487 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":126,"threads_started":1}
I20260807 02:26:03.436887 25187 tablet_service.cc:1467] Processing CreateTablet for tablet eec375ff15a34ef6abe207ed696fa476 (DEFAULT_TABLE table=test-table [id=a7205fd9136b407480948cc744835209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:26:03.436897 25327 tablet_service.cc:1467] Processing CreateTablet for tablet eec375ff15a34ef6abe207ed696fa476 (DEFAULT_TABLE table=test-table [id=a7205fd9136b407480948cc744835209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:26:03.436897 25047 tablet_service.cc:1467] Processing CreateTablet for tablet eec375ff15a34ef6abe207ed696fa476 (DEFAULT_TABLE table=test-table [id=a7205fd9136b407480948cc744835209]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:26:03.437958 25047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec375ff15a34ef6abe207ed696fa476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:03.437958 25327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec375ff15a34ef6abe207ed696fa476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:03.437968 25187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec375ff15a34ef6abe207ed696fa476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:03.444111 25414 tablet_bootstrap.cc:492] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f: Bootstrap starting.
I20260807 02:26:03.445591 25415 tablet_bootstrap.cc:492] T eec375ff15a34ef6abe207ed696fa476 P 9867ca656c6a42ee982a35324f0f14d7: Bootstrap starting.
I20260807 02:26:03.446135 25416 tablet_bootstrap.cc:492] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc: Bootstrap starting.
I20260807 02:26:03.446377 25414 tablet_bootstrap.cc:654] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:03.447389 25414 log.cc:824] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:03.447713 25415 tablet_bootstrap.cc:654] T eec375ff15a34ef6abe207ed696fa476 P 9867ca656c6a42ee982a35324f0f14d7: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:03.448733 25416 tablet_bootstrap.cc:654] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:03.449378 25415 log.cc:824] T eec375ff15a34ef6abe207ed696fa476 P 9867ca656c6a42ee982a35324f0f14d7: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:03.449503 25414 tablet_bootstrap.cc:492] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f: No bootstrap required, opened a new log
I20260807 02:26:03.449558 25416 log.cc:824] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:03.449714 25414 ts_tablet_manager.cc:1397] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 02:26:03.451171 25415 tablet_bootstrap.cc:492] T eec375ff15a34ef6abe207ed696fa476 P 9867ca656c6a42ee982a35324f0f14d7: No bootstrap required, opened a new log
I20260807 02:26:03.451371 25415 ts_tablet_manager.cc:1397] T eec375ff15a34ef6abe207ed696fa476 P 9867ca656c6a42ee982a35324f0f14d7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260807 02:26:03.452845 25414 raft_consensus.cc:348] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eee1d4dab1c4e18a667876cb422c1bc" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45225 } } peers { permanent_uuid: "9867ca656c6a42ee982a35324f0f14d7" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 34001 } } peers { permanent_uuid: "977a9133c2a54c77855b31e7b7f4311f" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 43307 } }
I20260807 02:26:03.453094 25414 raft_consensus.cc:374] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:03.453162 25414 raft_consensus.cc:729] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 977a9133c2a54c77855b31e7b7f4311f, State: Initialized, Role: FOLLOWER
I20260807 02:26:03.453670 25414 consensus_queue.cc:260] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eee1d4dab1c4e18a667876cb422c1bc" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45225 } } peers { permanent_uuid: "9867ca656c6a42ee982a35324f0f14d7" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 34001 } } peers { permanent_uuid: "977a9133c2a54c77855b31e7b7f4311f" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 43307 } }
I20260807 02:26:03.454393 25256 heartbeater.cc:503] Master 127.4.221.126:40099 was elected leader, sending a full tablet report...
I20260807 02:26:03.454407 25416 tablet_bootstrap.cc:492] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc: No bootstrap required, opened a new log
I20260807 02:26:03.454576 25416 ts_tablet_manager.cc:1397] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20260807 02:26:03.454504 25415 raft_consensus.cc:348] T eec375ff15a34ef6abe207ed696fa476 P 9867ca656c6a42ee982a35324f0f14d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eee1d4dab1c4e18a667876cb422c1bc" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45225 } } peers { permanent_uuid: "9867ca656c6a42ee982a35324f0f14d7" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 34001 } } peers { permanent_uuid: "977a9133c2a54c77855b31e7b7f4311f" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 43307 } }
I20260807 02:26:03.454711 25414 ts_tablet_manager.cc:1428] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 02:26:03.454722 25415 raft_consensus.cc:374] T eec375ff15a34ef6abe207ed696fa476 P 9867ca656c6a42ee982a35324f0f14d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:03.454768 25415 raft_consensus.cc:729] T eec375ff15a34ef6abe207ed696fa476 P 9867ca656c6a42ee982a35324f0f14d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9867ca656c6a42ee982a35324f0f14d7, State: Initialized, Role: FOLLOWER
I20260807 02:26:03.455205 25415 consensus_queue.cc:260] T eec375ff15a34ef6abe207ed696fa476 P 9867ca656c6a42ee982a35324f0f14d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eee1d4dab1c4e18a667876cb422c1bc" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45225 } } peers { permanent_uuid: "9867ca656c6a42ee982a35324f0f14d7" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 34001 } } peers { permanent_uuid: "977a9133c2a54c77855b31e7b7f4311f" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 43307 } }
I20260807 02:26:03.456087 25415 ts_tablet_manager.cc:1428] T eec375ff15a34ef6abe207ed696fa476 P 9867ca656c6a42ee982a35324f0f14d7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260807 02:26:03.456116 25115 heartbeater.cc:503] Master 127.4.221.126:40099 was elected leader, sending a full tablet report...
I20260807 02:26:03.458693 25416 raft_consensus.cc:348] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eee1d4dab1c4e18a667876cb422c1bc" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45225 } } peers { permanent_uuid: "9867ca656c6a42ee982a35324f0f14d7" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 34001 } } peers { permanent_uuid: "977a9133c2a54c77855b31e7b7f4311f" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 43307 } }
I20260807 02:26:03.458904 25416 raft_consensus.cc:374] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:03.458976 25416 raft_consensus.cc:729] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eee1d4dab1c4e18a667876cb422c1bc, State: Initialized, Role: FOLLOWER
I20260807 02:26:03.459362 25416 consensus_queue.cc:260] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eee1d4dab1c4e18a667876cb422c1bc" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45225 } } peers { permanent_uuid: "9867ca656c6a42ee982a35324f0f14d7" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 34001 } } peers { permanent_uuid: "977a9133c2a54c77855b31e7b7f4311f" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 43307 } }
I20260807 02:26:03.459964 25396 heartbeater.cc:503] Master 127.4.221.126:40099 was elected leader, sending a full tablet report...
I20260807 02:26:03.460224 25416 ts_tablet_manager.cc:1428] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
W20260807 02:26:03.626713 25397 tablet.cc:2369] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:26:03.663196 25257 tablet.cc:2369] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260807 02:26:03.699978 25117 tablet.cc:2369] T eec375ff15a34ef6abe207ed696fa476 P 9867ca656c6a42ee982a35324f0f14d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:26:03.826782 25420 raft_consensus.cc:482] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:26:03.827014 25420 raft_consensus.cc:504] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eee1d4dab1c4e18a667876cb422c1bc" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45225 } } peers { permanent_uuid: "9867ca656c6a42ee982a35324f0f14d7" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 34001 } } peers { permanent_uuid: "977a9133c2a54c77855b31e7b7f4311f" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 43307 } }
I20260807 02:26:03.828366 25420 leader_election.cc:288] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eee1d4dab1c4e18a667876cb422c1bc (127.4.221.67:45225), 9867ca656c6a42ee982a35324f0f14d7 (127.4.221.65:34001)
I20260807 02:26:03.832494 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.828735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:45461 (local address 127.4.221.65:34001)
0807 02:26:03.828890 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:03.828895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:03.829071 (+ 176us) server_negotiation.cc:407] Connection header received
0807 02:26:03.829241 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:03.829244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:03.829287 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:03.829371 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:26:03.829703 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.830307 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.831535 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.831922 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.832050 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.832101 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:03.832302 (+ 201us) server_negotiation.cc:299] Negotiation successful
0807 02:26:03.832367 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 02:26:03.832516 25424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.828662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:34001 (local address 127.4.221.66:45461)
0807 02:26:03.828966 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:03.828977 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:03.829115 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:03.829392 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:03.829395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:03.829410 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:26:03.829563 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:03.829569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.830425 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.830429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:03.831412 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:03.831421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.832050 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.832054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:03.832149 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.832174 (+ 25us) client_negotiation.cc:769] Sending connection context
0807 02:26:03.832283 (+ 109us) client_negotiation.cc:240] Negotiation successful
0807 02:26:03.832355 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":87,"spinlock_wait_cycles":896,"thread_start_us":66,"threads_started":1}
I20260807 02:26:03.832635 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.828744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:40881 (local address 127.4.221.67:45225)
0807 02:26:03.828884 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:03.828890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:03.829055 (+ 165us) server_negotiation.cc:407] Connection header received
0807 02:26:03.829252 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:03.829255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:03.829300 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:03.829397 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:26:03.829752 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.830313 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.831641 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.832050 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.832195 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.832252 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:03.832426 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 02:26:03.832501 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260807 02:26:03.832893 25423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.828573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:45225 (local address 127.4.221.66:40881)
0807 02:26:03.828928 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:03.828946 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:03.829099 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:03.829411 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:03.829415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:03.829457 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:26:03.829622 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:03.829628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.830438 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.830443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:03.831412 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:03.831422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.832185 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.832189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:03.832293 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.832314 (+ 21us) client_negotiation.cc:769] Sending connection context
0807 02:26:03.832411 (+ 97us) client_negotiation.cc:240] Negotiation successful
0807 02:26:03.832480 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":23,"spinlock_wait_cycles":25216,"thread_start_us":103,"threads_started":1}
I20260807 02:26:03.833307 25067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec375ff15a34ef6abe207ed696fa476" candidate_uuid: "977a9133c2a54c77855b31e7b7f4311f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9867ca656c6a42ee982a35324f0f14d7" is_pre_election: true
I20260807 02:26:03.833494 25348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec375ff15a34ef6abe207ed696fa476" candidate_uuid: "977a9133c2a54c77855b31e7b7f4311f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eee1d4dab1c4e18a667876cb422c1bc" is_pre_election: true
I20260807 02:26:03.833642 25067 raft_consensus.cc:2454] T eec375ff15a34ef6abe207ed696fa476 P 9867ca656c6a42ee982a35324f0f14d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 977a9133c2a54c77855b31e7b7f4311f in term 0.
I20260807 02:26:03.833803 25348 raft_consensus.cc:2454] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 977a9133c2a54c77855b31e7b7f4311f in term 0.
I20260807 02:26:03.834242 25142 leader_election.cc:302] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 977a9133c2a54c77855b31e7b7f4311f, 9867ca656c6a42ee982a35324f0f14d7; no voters:
I20260807 02:26:03.834542 25420 raft_consensus.cc:2781] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:26:03.834628 25420 raft_consensus.cc:482] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:26:03.834673 25420 raft_consensus.cc:3037] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:03.835718 25420 raft_consensus.cc:504] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eee1d4dab1c4e18a667876cb422c1bc" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45225 } } peers { permanent_uuid: "9867ca656c6a42ee982a35324f0f14d7" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 34001 } } peers { permanent_uuid: "977a9133c2a54c77855b31e7b7f4311f" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 43307 } }
I20260807 02:26:03.836086 25420 leader_election.cc:288] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [CANDIDATE]: Term 1 election: Requested vote from peers 0eee1d4dab1c4e18a667876cb422c1bc (127.4.221.67:45225), 9867ca656c6a42ee982a35324f0f14d7 (127.4.221.65:34001)
I20260807 02:26:03.836445 25067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec375ff15a34ef6abe207ed696fa476" candidate_uuid: "977a9133c2a54c77855b31e7b7f4311f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9867ca656c6a42ee982a35324f0f14d7"
I20260807 02:26:03.836546 25348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec375ff15a34ef6abe207ed696fa476" candidate_uuid: "977a9133c2a54c77855b31e7b7f4311f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eee1d4dab1c4e18a667876cb422c1bc"
I20260807 02:26:03.836563 25067 raft_consensus.cc:3037] T eec375ff15a34ef6abe207ed696fa476 P 9867ca656c6a42ee982a35324f0f14d7 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:03.836692 25348 raft_consensus.cc:3037] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:03.837610 25067 raft_consensus.cc:2454] T eec375ff15a34ef6abe207ed696fa476 P 9867ca656c6a42ee982a35324f0f14d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 977a9133c2a54c77855b31e7b7f4311f in term 1.
I20260807 02:26:03.837787 25348 raft_consensus.cc:2454] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 977a9133c2a54c77855b31e7b7f4311f in term 1.
I20260807 02:26:03.837957 25142 leader_election.cc:302] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 977a9133c2a54c77855b31e7b7f4311f, 9867ca656c6a42ee982a35324f0f14d7; no voters:
I20260807 02:26:03.838162 25420 raft_consensus.cc:2781] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:03.838367 25420 raft_consensus.cc:686] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [term 1 LEADER]: Becoming Leader. State: Replica: 977a9133c2a54c77855b31e7b7f4311f, State: Running, Role: LEADER
I20260807 02:26:03.838727 25420 consensus_queue.cc:237] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eee1d4dab1c4e18a667876cb422c1bc" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45225 } } peers { permanent_uuid: "9867ca656c6a42ee982a35324f0f14d7" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 34001 } } peers { permanent_uuid: "977a9133c2a54c77855b31e7b7f4311f" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 43307 } }
I20260807 02:26:03.841809 24916 catalog_manager.cc:5495] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f reported cstate change: term changed from 0 to 1, leader changed from <none> to 977a9133c2a54c77855b31e7b7f4311f (127.4.221.66). New cstate: current_term: 1 leader_uuid: "977a9133c2a54c77855b31e7b7f4311f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eee1d4dab1c4e18a667876cb422c1bc" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9867ca656c6a42ee982a35324f0f14d7" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 34001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "977a9133c2a54c77855b31e7b7f4311f" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 43307 } health_report { overall_health: HEALTHY } } }
I20260807 02:26:03.883406 25413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.879784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.4.221.66:43307)
0807 02:26:03.879939 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:03.879943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:03.879957 (+ 14us) server_negotiation.cc:407] Connection header received
0807 02:26:03.880043 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:03.880045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:03.880105 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:03.880207 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:26:03.880432 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.880993 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.881872 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:03.882092 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:03.882184 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:03.882559 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:26:03.882976 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:26:03.883072 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:03.883220 (+ 148us) server_negotiation.cc:299] Negotiation successful
0807 02:26:03.883292 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:26:03.889588 25067 raft_consensus.cc:1261] T eec375ff15a34ef6abe207ed696fa476 P 9867ca656c6a42ee982a35324f0f14d7 [term 1 FOLLOWER]: Refusing update from remote peer 977a9133c2a54c77855b31e7b7f4311f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:26:03.889601 25348 raft_consensus.cc:1261] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc [term 1 FOLLOWER]: Refusing update from remote peer 977a9133c2a54c77855b31e7b7f4311f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:26:03.890347 25420 consensus_queue.cc:1035] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9867ca656c6a42ee982a35324f0f14d7" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 34001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:26:03.890578 25425 consensus_queue.cc:1035] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eee1d4dab1c4e18a667876cb422c1bc" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 45225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:26:03.897925 25430 mvcc.cc:205] Tried to move back new op lower bound from 7315740933683621888 to 7315740933485981696. Current Snapshot: MvccSnapshot[applied={T|T < 7315740933683621888}]
I20260807 02:26:03.906883 24917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:26:03.907083 24917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:26:03.909801 24917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3aa6ae59c5346a6ba728d44268a8668: Sending DeleteTablet for 3 replicas of tablet eec375ff15a34ef6abe207ed696fa476
I20260807 02:26:03.910665 25327 tablet_service.cc:1514] Processing DeleteTablet for tablet eec375ff15a34ef6abe207ed696fa476 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:26:03 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:42350
I20260807 02:26:03.910699 25047 tablet_service.cc:1514] Processing DeleteTablet for tablet eec375ff15a34ef6abe207ed696fa476 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:26:03 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:35900
I20260807 02:26:03.910779 25187 tablet_service.cc:1514] Processing DeleteTablet for tablet eec375ff15a34ef6abe207ed696fa476 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:26:03 UTC) from {username='kudu', principal='kudu/127.4.221.126@KRBTEST.COM'} at 127.0.0.1:41096
I20260807 02:26:03.911480 25439 tablet_replica.cc:330] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f: stopping tablet replica
I20260807 02:26:03.911809 25439 raft_consensus.cc:2229] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:26:03.911876 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 24982
I20260807 02:26:03.912186 25439 raft_consensus.cc:2258] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:26:03.914129 25439 ts_tablet_manager.cc:1905] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:26:03.917443 25439 ts_tablet_manager.cc:1918] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:26:03.917569 25439 log.cc:1196] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/eec375ff15a34ef6abe207ed696fa476
I20260807 02:26:03.917912 25439 ts_tablet_manager.cc:1939] T eec375ff15a34ef6abe207ed696fa476 P 977a9133c2a54c77855b31e7b7f4311f: Deleting consensus metadata
I20260807 02:26:03.919075 24901 catalog_manager.cc:4841] TS 977a9133c2a54c77855b31e7b7f4311f (127.4.221.66:43307): tablet eec375ff15a34ef6abe207ed696fa476 (table test-table [id=a7205fd9136b407480948cc744835209]) successfully deleted
I20260807 02:26:03.919464 25440 tablet_replica.cc:330] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc: stopping tablet replica
I20260807 02:26:03.919790 25440 raft_consensus.cc:2229] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:26:03.920120 25440 raft_consensus.cc:2258] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:26:03.921555 25440 ts_tablet_manager.cc:1905] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:26:03.923938 25440 ts_tablet_manager.cc:1918] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:26:03.924052 25440 log.cc:1196] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/eec375ff15a34ef6abe207ed696fa476
I20260807 02:26:03.924412 25440 ts_tablet_manager.cc:1939] T eec375ff15a34ef6abe207ed696fa476 P 0eee1d4dab1c4e18a667876cb422c1bc: Deleting consensus metadata
W20260807 02:26:03.924515 24904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.221.65:34001 (error 108)
I20260807 02:26:03.925359 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 25121
I20260807 02:26:03.925570 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:03.925214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:34001 (local address 127.0.0.1:35912)
0807 02:26:03.925396 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:03.925468 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:34001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260807 02:26:03.927013 24902 catalog_manager.cc:4841] TS 0eee1d4dab1c4e18a667876cb422c1bc (127.4.221.67:45225): tablet eec375ff15a34ef6abe207ed696fa476 (table test-table [id=a7205fd9136b407480948cc744835209]) successfully deleted
W20260807 02:26:03.927610 24904 catalog_manager.cc:4584] TS 9867ca656c6a42ee982a35324f0f14d7 (127.4.221.65:34001): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eec375ff15a34ef6abe207ed696fa476: Network error: Client connection negotiation failed: client connection to 127.4.221.65:34001: connect: Connection refused (error 111)
I20260807 02:26:03.932312 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 25261
I20260807 02:26:03.938335 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 24883
2026-08-07T02:26:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2682 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-07T02:26:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:03Z Disabled control of system clock
I20260807 02:26:03.972488 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44985
--webserver_interface=127.4.221.126
--webserver_port=0
--builtin_ntp_servers=127.4.221.84:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:44985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 02:26:04.089488 25445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:04.089775 25445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:04.089812 25445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:04.093731 25445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:04.093783 25445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:04.093799 25445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:04.093868 25445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:04.093899 25445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:04.098585 25445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:44985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:04.099560 25445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:04.100906 25445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 02:26:04.109380 25445 server_base.cc:1034] running on GCE node
W20260807 02:26:04.109354 25451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:04.109344 25450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:04.109321 25453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:04.110248 25445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:04.111218 25445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:04.112411 25445 hybrid_clock.cc:648] HybridClock initialized: now 1786069564112395 us; error 30 us; skew 500 ppm
I20260807 02:26:04.120311 25445 webserver.cc:460] Webserver started at http://127.4.221.126:34021/ using document root <none> and password file <none>
I20260807 02:26:04.120879 25445 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:04.120970 25445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:04.121202 25445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:04.123338 25445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "e221f758bbe24c0e93efd020be9d9a0b"
format_stamp: "Formatted at 2026-08-07 02:26:04 on dist-test-slave-69rg"
server_key: "b5e278f3d7eb476170a469e6ef7080ee"
server_key_iv: "953efe88676ff5ae858248dc79cdc0d5"
server_key_version: "encryptionkey@0"
I20260807 02:26:04.123905 25445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "e221f758bbe24c0e93efd020be9d9a0b"
format_stamp: "Formatted at 2026-08-07 02:26:04 on dist-test-slave-69rg"
server_key: "b5e278f3d7eb476170a469e6ef7080ee"
server_key_iv: "953efe88676ff5ae858248dc79cdc0d5"
server_key_version: "encryptionkey@0"
I20260807 02:26:04.127584 25445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 02:26:04.130209 25464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:04.131379 25445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:26:04.131534 25445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "e221f758bbe24c0e93efd020be9d9a0b"
format_stamp: "Formatted at 2026-08-07 02:26:04 on dist-test-slave-69rg"
server_key: "b5e278f3d7eb476170a469e6ef7080ee"
server_key_iv: "953efe88676ff5ae858248dc79cdc0d5"
server_key_version: "encryptionkey@0"
I20260807 02:26:04.131651 25445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:04.146561 25445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:04.147176 25445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:04.147369 25445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:04.154624 25445 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:44985
I20260807 02:26:04.154635 25520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:44985 every 8 connection(s)
I20260807 02:26:04.155544 25445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:04.158561 25521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:04.160733 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 25445
I20260807 02:26:04.160887 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:04.161181 4981 external_mini_cluster.cc:1428] Setting key 9fc852d9fdc16d4b5a8e43ccc55aaac4
I20260807 02:26:04.164117 25521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b: Bootstrap starting.
I20260807 02:26:04.166532 25521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:04.167301 25521 log.cc:824] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:04.169368 25521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b: No bootstrap required, opened a new log
I20260807 02:26:04.170097 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.162519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.4.221.126:44985)
0807 02:26:04.163071 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:04.163086 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:04.163121 (+ 35us) server_negotiation.cc:407] Connection header received
0807 02:26:04.163741 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:04.163756 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:04.164108 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:04.164411 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:04.164725 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.165639 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.166855 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.167289 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.167449 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.168391 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:04.168396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:04.168407 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:04.168755 (+ 348us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:04.168805 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:04.168825 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:04.168901 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:04.169069 (+ 168us) server_negotiation.cc:299] Negotiation successful
0807 02:26:04.169350 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":119,"threads_started":1}
I20260807 02:26:04.172737 25521 raft_consensus.cc:348] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e221f758bbe24c0e93efd020be9d9a0b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44985 } }
I20260807 02:26:04.172935 25521 raft_consensus.cc:374] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:04.173012 25521 raft_consensus.cc:729] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e221f758bbe24c0e93efd020be9d9a0b, State: Initialized, Role: FOLLOWER
I20260807 02:26:04.173420 25521 consensus_queue.cc:260] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e221f758bbe24c0e93efd020be9d9a0b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44985 } }
I20260807 02:26:04.173574 25521 raft_consensus.cc:388] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:04.173669 25521 raft_consensus.cc:482] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:04.173786 25521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:04.174676 25521 raft_consensus.cc:504] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e221f758bbe24c0e93efd020be9d9a0b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44985 } }
I20260807 02:26:04.175006 25521 leader_election.cc:302] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e221f758bbe24c0e93efd020be9d9a0b; no voters:
I20260807 02:26:04.175307 25521 leader_election.cc:288] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:04.175443 25527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:04.175695 25527 raft_consensus.cc:686] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [term 1 LEADER]: Becoming Leader. State: Replica: e221f758bbe24c0e93efd020be9d9a0b, State: Running, Role: LEADER
I20260807 02:26:04.176002 25527 consensus_queue.cc:237] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e221f758bbe24c0e93efd020be9d9a0b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44985 } }
I20260807 02:26:04.176476 25521 sys_catalog.cc:564] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:04.177629 25528 sys_catalog.cc:455] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e221f758bbe24c0e93efd020be9d9a0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e221f758bbe24c0e93efd020be9d9a0b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44985 } } }
I20260807 02:26:04.177862 25528 sys_catalog.cc:458] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:04.178174 25529 sys_catalog.cc:455] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e221f758bbe24c0e93efd020be9d9a0b. Latest consensus state: current_term: 1 leader_uuid: "e221f758bbe24c0e93efd020be9d9a0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e221f758bbe24c0e93efd020be9d9a0b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44985 } } }
I20260807 02:26:04.178298 25529 sys_catalog.cc:458] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:04.179031 25537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:04.181527 25537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:04.186936 25537 catalog_manager.cc:1347] Generated new cluster ID: 25a865c9f4274cc1b5514bd9a6856c89
I20260807 02:26:04.187016 25537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:04.208400 25537 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:04.209600 25537 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:04.216737 25537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b: Generated new TSK 0
I20260807 02:26:04.217499 25537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:04.221400 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.65:0
--local_ip_for_outbound_sockets=127.4.221.65
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44985
--builtin_ntp_servers=127.4.221.84:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260807 02:26:04.334084 25546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:04.334415 25546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:04.334501 25546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:04.338718 25546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:04.338789 25546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:04.338878 25546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.65
I20260807 02:26:04.343482 25546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.221.65
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.65
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:04.344498 25546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:04.345871 25546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:04.355361 25551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:04.355335 25555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:04.355502 25546 server_base.cc:1034] running on GCE node
W20260807 02:26:04.355322 25553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:04.356040 25546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:04.356700 25546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:04.357897 25546 hybrid_clock.cc:648] HybridClock initialized: now 1786069564357883 us; error 24 us; skew 500 ppm
I20260807 02:26:04.359745 25546 webserver.cc:460] Webserver started at http://127.4.221.65:38515/ using document root <none> and password file <none>
I20260807 02:26:04.360299 25546 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:04.360363 25546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:04.360623 25546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:04.362676 25546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/data/instance:
uuid: "9c464a1082f04ffbacb5dabecc068688"
format_stamp: "Formatted at 2026-08-07 02:26:04 on dist-test-slave-69rg"
server_key: "b6563d162ad5954b13bb98f7248faa5c"
server_key_iv: "4b35a758707d415f0ccbf4b03e0147c2"
server_key_version: "encryptionkey@0"
I20260807 02:26:04.363229 25546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance:
uuid: "9c464a1082f04ffbacb5dabecc068688"
format_stamp: "Formatted at 2026-08-07 02:26:04 on dist-test-slave-69rg"
server_key: "b6563d162ad5954b13bb98f7248faa5c"
server_key_iv: "4b35a758707d415f0ccbf4b03e0147c2"
server_key_version: "encryptionkey@0"
I20260807 02:26:04.366847 25546 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:26:04.369220 25561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:04.370496 25546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:26:04.370635 25546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/wal
uuid: "9c464a1082f04ffbacb5dabecc068688"
format_stamp: "Formatted at 2026-08-07 02:26:04 on dist-test-slave-69rg"
server_key: "b6563d162ad5954b13bb98f7248faa5c"
server_key_iv: "4b35a758707d415f0ccbf4b03e0147c2"
server_key_version: "encryptionkey@0"
I20260807 02:26:04.370750 25546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:04.389617 25546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:04.390399 25546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:04.390604 25546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:04.391206 25546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:26:04.392158 25546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:26:04.392230 25546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:04.392302 25546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:26:04.392356 25546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:04.401280 25546 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.65:33485
I20260807 02:26:04.401353 25681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.65:33485 every 8 connection(s)
I20260807 02:26:04.402261 25546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/data/info.pb
I20260807 02:26:04.409013 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 25546
I20260807 02:26:04.409171 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-0/wal/instance
I20260807 02:26:04.409458 4981 external_mini_cluster.cc:1428] Setting key 9c7c173c00ffbf613991b2dd0ea58076
I20260807 02:26:04.411381 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.66:0
--local_ip_for_outbound_sockets=127.4.221.66
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44985
--builtin_ntp_servers=127.4.221.84:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 02:26:04.411422 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.404039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.65:49811 (local address 127.4.221.126:44985)
0807 02:26:04.404170 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:04.404175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:04.405145 (+ 970us) server_negotiation.cc:407] Connection header received
0807 02:26:04.406619 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:04.406623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:04.406684 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:04.406788 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:04.407631 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.408178 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.409380 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.409657 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.409767 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.410916 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:04.410920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:04.410922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:04.410935 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:04.410956 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:04.410973 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:04.411049 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:04.411220 (+ 171us) server_negotiation.cc:299] Negotiation successful
0807 02:26:04.411282 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 02:26:04.412245 25684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.404369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44985 (local address 127.4.221.65:49811)
0807 02:26:04.404987 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:04.405026 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:04.406404 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:04.406953 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:04.406960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:04.407037 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:04.407486 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.407498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.408310 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.408318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.409249 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.409261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.409849 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.409858 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.410018 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.410660 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:26:04.410661 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:04.410701 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:26:04.410760 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:04.411064 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:04.411076 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:04.411086 (+ 10us) client_negotiation.cc:769] Sending connection context
0807 02:26:04.411207 (+ 121us) client_negotiation.cc:240] Negotiation successful
0807 02:26:04.411477 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":119,"threads_started":1}
I20260807 02:26:04.413642 25682 heartbeater.cc:347] Connected to a master server at 127.4.221.126:44985
I20260807 02:26:04.413926 25682 heartbeater.cc:464] Registering TS with master...
I20260807 02:26:04.414520 25682 heartbeater.cc:511] Master 127.4.221.126:44985 requested a full tablet report, sending...
I20260807 02:26:04.416275 25482 ts_manager.cc:194] Registered new tserver with Master: 9c464a1082f04ffbacb5dabecc068688 (127.4.221.65:33485)
I20260807 02:26:04.418082 25482 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.221.65:49811
W20260807 02:26:04.523201 25685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:04.523443 25685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:04.523481 25685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:04.527360 25685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:04.527413 25685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:04.527467 25685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.66
I20260807 02:26:04.532094 25685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.221.66
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.66
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:04.533174 25685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:04.534523 25685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:04.544490 25691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:04.544600 25694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:04.544492 25692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:04.544798 25685 server_base.cc:1034] running on GCE node
I20260807 02:26:04.545282 25685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:04.545981 25685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:04.547202 25685 hybrid_clock.cc:648] HybridClock initialized: now 1786069564547181 us; error 32 us; skew 500 ppm
I20260807 02:26:04.549170 25685 webserver.cc:460] Webserver started at http://127.4.221.66:35173/ using document root <none> and password file <none>
I20260807 02:26:04.549747 25685 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:04.549810 25685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:04.550071 25685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:04.552248 25685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/data/instance:
uuid: "f0c063e6e5d14f8195909de92231e4f8"
format_stamp: "Formatted at 2026-08-07 02:26:04 on dist-test-slave-69rg"
server_key: "44a036c209ef267382e373a9c5b09fed"
server_key_iv: "d6d6b9cffec0e9209dbe0648a6947491"
server_key_version: "encryptionkey@0"
I20260807 02:26:04.552763 25685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance:
uuid: "f0c063e6e5d14f8195909de92231e4f8"
format_stamp: "Formatted at 2026-08-07 02:26:04 on dist-test-slave-69rg"
server_key: "44a036c209ef267382e373a9c5b09fed"
server_key_iv: "d6d6b9cffec0e9209dbe0648a6947491"
server_key_version: "encryptionkey@0"
I20260807 02:26:04.556414 25685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260807 02:26:04.558796 25701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:04.559952 25685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:26:04.560048 25685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/wal
uuid: "f0c063e6e5d14f8195909de92231e4f8"
format_stamp: "Formatted at 2026-08-07 02:26:04 on dist-test-slave-69rg"
server_key: "44a036c209ef267382e373a9c5b09fed"
server_key_iv: "d6d6b9cffec0e9209dbe0648a6947491"
server_key_version: "encryptionkey@0"
I20260807 02:26:04.560138 25685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:04.580241 25685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:04.580914 25685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:04.581073 25685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:04.581629 25685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:26:04.582615 25685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:26:04.582664 25685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:04.582703 25685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:26:04.582719 25685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:04.591737 25685 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.66:33525
I20260807 02:26:04.591764 25822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.66:33525 every 8 connection(s)
I20260807 02:26:04.592630 25685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/data/info.pb
I20260807 02:26:04.599356 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 25685
I20260807 02:26:04.599465 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/wal/instance
I20260807 02:26:04.599784 4981 external_mini_cluster.cc:1428] Setting key 6e8a1ce823c50c59a8c95983ef9ab5c7
I20260807 02:26:04.601752 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.221.67:0
--local_ip_for_outbound_sockets=127.4.221.67
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44985
--builtin_ntp_servers=127.4.221.84:44297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260807 02:26:04.602241 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.594494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:49445 (local address 127.4.221.126:44985)
0807 02:26:04.594641 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:04.594645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:04.595691 (+ 1046us) server_negotiation.cc:407] Connection header received
0807 02:26:04.597073 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:04.597076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:04.597124 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:04.597249 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:04.598280 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.598822 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.600131 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.600359 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.600474 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.601628 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:04.601632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:04.601635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:04.601651 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:04.601674 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:04.601682 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:04.601764 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:04.602036 (+ 272us) server_negotiation.cc:299] Negotiation successful
0807 02:26:04.602100 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:26:04.603096 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.594879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44985 (local address 127.4.221.66:49445)
0807 02:26:04.595536 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:04.595576 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:04.596836 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:04.597436 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:04.597443 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:04.597523 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:04.598077 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.598092 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.598944 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.598947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.599998 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.600009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.600517 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.600521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.600693 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.601351 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:26:04.601353 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:04.601394 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:26:04.601458 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:04.601768 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:04.601781 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:04.601788 (+ 7us) client_negotiation.cc:769] Sending connection context
0807 02:26:04.602033 (+ 245us) client_negotiation.cc:240] Negotiation successful
0807 02:26:04.602325 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":125,"threads_started":1}
I20260807 02:26:04.604287 25823 heartbeater.cc:347] Connected to a master server at 127.4.221.126:44985
I20260807 02:26:04.604528 25823 heartbeater.cc:464] Registering TS with master...
I20260807 02:26:04.605037 25823 heartbeater.cc:511] Master 127.4.221.126:44985 requested a full tablet report, sending...
I20260807 02:26:04.606194 25482 ts_manager.cc:194] Registered new tserver with Master: f0c063e6e5d14f8195909de92231e4f8 (127.4.221.66:33525)
I20260807 02:26:04.607016 25482 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.221.66:49445
W20260807 02:26:04.713749 25827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:04.714020 25827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:04.714059 25827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:04.718307 25827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:04.718355 25827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:04.718410 25827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.221.67
I20260807 02:26:04.723026 25827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:44297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.221.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.221.67
--webserver_port=0
--tserver_master_addrs=127.4.221.126:44985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.221.67
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:04.723969 25827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:04.725276 25827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:04.734671 25835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:04.734678 25832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:04.734689 25833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:04.734978 25827 server_base.cc:1034] running on GCE node
I20260807 02:26:04.735615 25827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:04.736316 25827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:04.737488 25827 hybrid_clock.cc:648] HybridClock initialized: now 1786069564737458 us; error 39 us; skew 500 ppm
I20260807 02:26:04.739534 25827 webserver.cc:460] Webserver started at http://127.4.221.67:35465/ using document root <none> and password file <none>
I20260807 02:26:04.740082 25827 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:04.740206 25827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:04.740468 25827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:04.742544 25827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/data/instance:
uuid: "e4343fe518b34c46a90d7dbeece7f96c"
format_stamp: "Formatted at 2026-08-07 02:26:04 on dist-test-slave-69rg"
server_key: "4453ca0d0e954401f0854e158fbfe521"
server_key_iv: "34e9e76009d8437552ea2cb9406a55ca"
server_key_version: "encryptionkey@0"
I20260807 02:26:04.743104 25827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance:
uuid: "e4343fe518b34c46a90d7dbeece7f96c"
format_stamp: "Formatted at 2026-08-07 02:26:04 on dist-test-slave-69rg"
server_key: "4453ca0d0e954401f0854e158fbfe521"
server_key_iv: "34e9e76009d8437552ea2cb9406a55ca"
server_key_version: "encryptionkey@0"
I20260807 02:26:04.746804 25827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260807 02:26:04.749177 25842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:04.750465 25827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 02:26:04.750608 25827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/wal
uuid: "e4343fe518b34c46a90d7dbeece7f96c"
format_stamp: "Formatted at 2026-08-07 02:26:04 on dist-test-slave-69rg"
server_key: "4453ca0d0e954401f0854e158fbfe521"
server_key_iv: "34e9e76009d8437552ea2cb9406a55ca"
server_key_version: "encryptionkey@0"
I20260807 02:26:04.750722 25827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:04.767704 25827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:04.768359 25827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:04.768548 25827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:04.769136 25827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260807 02:26:04.770166 25827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260807 02:26:04.770238 25827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:04.770306 25827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260807 02:26:04.770356 25827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:04.779569 25827 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.67:33299
I20260807 02:26:04.779593 25959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.67:33299 every 8 connection(s)
I20260807 02:26:04.780617 25827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/data/info.pb
I20260807 02:26:04.789929 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.782576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.67:33693 (local address 127.4.221.126:44985)
0807 02:26:04.782752 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:04.782756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:04.783748 (+ 992us) server_negotiation.cc:407] Connection header received
0807 02:26:04.785150 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:04.785154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:04.785212 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:04.785286 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:04.786178 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.786725 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.787895 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.788125 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.788234 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.789369 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:04.789373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:04.789376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:04.789389 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:04.789410 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:04.789426 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:04.789506 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:04.789686 (+ 180us) server_negotiation.cc:299] Negotiation successful
0807 02:26:04.789747 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260807 02:26:04.790374 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 25827
I20260807 02:26:04.790477 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/wal/instance
I20260807 02:26:04.790807 4981 external_mini_cluster.cc:1428] Setting key 6e79e02724bf6e2bdaaf643fa595cf0b
I20260807 02:26:04.790934 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.782958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:44985 (local address 127.4.221.67:33693)
0807 02:26:04.783589 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:04.783631 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:04.784933 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:04.785443 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:04.785450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:04.785530 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:04.786022 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.786034 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.786861 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.786864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.787765 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.787774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.788284 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.788288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.788445 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.789104 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0807 02:26:04.789105 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:04.789149 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0807 02:26:04.789208 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:04.789508 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:04.789520 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:04.789529 (+ 9us) client_negotiation.cc:769] Sending connection context
0807 02:26:04.789660 (+ 131us) client_negotiation.cc:240] Negotiation successful
0807 02:26:04.789949 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":119,"threads_started":1}
I20260807 02:26:04.792121 25960 heartbeater.cc:347] Connected to a master server at 127.4.221.126:44985
I20260807 02:26:04.792382 25960 heartbeater.cc:464] Registering TS with master...
I20260807 02:26:04.792939 25960 heartbeater.cc:511] Master 127.4.221.126:44985 requested a full tablet report, sending...
I20260807 02:26:04.793952 25481 ts_manager.cc:194] Registered new tserver with Master: e4343fe518b34c46a90d7dbeece7f96c (127.4.221.67:33299)
I20260807 02:26:04.794544 25481 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.221.67:33693
I20260807 02:26:04.804522 4981 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260807 02:26:04.810447 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.806546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.4.221.126:44985)
0807 02:26:04.806724 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:04.806728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:04.806884 (+ 156us) server_negotiation.cc:407] Connection header received
0807 02:26:04.807028 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:04.807031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:04.807073 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:04.807193 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:04.807481 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.807981 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.808826 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.809054 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.809164 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.810003 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:04.810006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:04.810009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:04.810021 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:04.810039 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:04.810056 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:04.810135 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:04.810229 (+ 94us) server_negotiation.cc:299] Negotiation successful
0807 02:26:04.810305 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260807 02:26:04.813158 25481 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260807 02:26:04.816434 25481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260807 02:26:04.830973 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.824946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33485 (local address 127.0.0.1:44682)
0807 02:26:04.825524 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:04.825542 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:04.825652 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:04.826439 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:04.826445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:04.826481 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:26:04.826931 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.826938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.828368 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.828371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.829472 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.829482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.830411 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.830415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.830519 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.830555 (+ 36us) client_negotiation.cc:769] Sending connection context
0807 02:26:04.830680 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 02:26:04.830798 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":28,"spinlock_wait_cycles":5248,"thread_start_us":102,"threads_started":1}
I20260807 02:26:04.831204 25978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.825693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.4.221.66:33525)
0807 02:26:04.826182 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:04.826194 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:04.826218 (+ 24us) server_negotiation.cc:407] Connection header received
0807 02:26:04.826293 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:04.826299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:04.826486 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:04.826650 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:26:04.826866 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.827894 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.829711 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.830581 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.830700 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.830890 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:04.830943 (+ 53us) server_negotiation.cc:299] Negotiation successful
0807 02:26:04.831046 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":184,"threads_started":1}
I20260807 02:26:04.831437 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.825435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44682 (local address 127.4.221.65:33485)
0807 02:26:04.826037 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:04.826047 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:04.826066 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:26:04.826130 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:04.826145 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:04.826285 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:04.827064 (+ 779us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:26:04.827218 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.828257 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.829599 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.830285 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.830410 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.830610 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:04.831172 (+ 562us) server_negotiation.cc:299] Negotiation successful
0807 02:26:04.831271 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":133,"threads_started":1}
I20260807 02:26:04.831626 25973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.824827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37868 (local address 127.4.221.67:33299)
0807 02:26:04.825935 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:04.825948 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:04.825974 (+ 26us) server_negotiation.cc:407] Connection header received
0807 02:26:04.826054 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:04.826060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:04.826256 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:04.826584 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:26:04.826836 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.827966 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.829851 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.830584 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.831090 (+ 506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.831271 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:04.831327 (+ 56us) server_negotiation.cc:299] Negotiation successful
0807 02:26:04.831437 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":969,"thread_start_us":140,"threads_started":1}
I20260807 02:26:04.831770 25972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.824614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:33299 (local address 127.0.0.1:37868)
0807 02:26:04.824961 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:04.825013 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:04.825136 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:04.826439 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:04.826445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:04.826483 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:26:04.826683 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.826693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.828126 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.828129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.829704 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.829712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.830728 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.830732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.830821 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.830843 (+ 22us) client_negotiation.cc:769] Sending connection context
0807 02:26:04.830899 (+ 56us) client_negotiation.cc:240] Negotiation successful
0807 02:26:04.830973 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":214,"spinlock_wait_cycles":7424,"thread_start_us":90,"threads_started":1}
I20260807 02:26:04.831034 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.825372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:33525 (local address 127.0.0.1:58510)
0807 02:26:04.825591 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:04.825606 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:04.825699 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:04.826615 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:04.826618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:04.826637 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:26:04.826734 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.826741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.828039 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.828043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.829568 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.829578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.830701 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.830705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.830783 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.830803 (+ 20us) client_negotiation.cc:769] Sending connection context
0807 02:26:04.830862 (+ 59us) client_negotiation.cc:240] Negotiation successful
0807 02:26:04.830918 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":151,"thread_start_us":84,"threads_started":1}
I20260807 02:26:04.834476 25752 tablet_service.cc:1467] Processing CreateTablet for tablet 3c162c4c4360481984c072a27f84ce0b (DEFAULT_TABLE table=test-table [id=ccd03d5db35a427588f32da5c257e829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:26:04.834474 25891 tablet_service.cc:1467] Processing CreateTablet for tablet 3c162c4c4360481984c072a27f84ce0b (DEFAULT_TABLE table=test-table [id=ccd03d5db35a427588f32da5c257e829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:26:04.834474 25614 tablet_service.cc:1467] Processing CreateTablet for tablet 3c162c4c4360481984c072a27f84ce0b (DEFAULT_TABLE table=test-table [id=ccd03d5db35a427588f32da5c257e829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260807 02:26:04.835551 25614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c162c4c4360481984c072a27f84ce0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:04.835551 25891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c162c4c4360481984c072a27f84ce0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:04.835537 25752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c162c4c4360481984c072a27f84ce0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:04.842028 25979 tablet_bootstrap.cc:492] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8: Bootstrap starting.
I20260807 02:26:04.843259 25980 tablet_bootstrap.cc:492] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c: Bootstrap starting.
I20260807 02:26:04.844874 25979 tablet_bootstrap.cc:654] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:04.844887 25980 tablet_bootstrap.cc:654] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:04.845588 25979 log.cc:824] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:04.845587 25980 log.cc:824] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:04.846262 25982 tablet_bootstrap.cc:492] T 3c162c4c4360481984c072a27f84ce0b P 9c464a1082f04ffbacb5dabecc068688: Bootstrap starting.
I20260807 02:26:04.847694 25980 tablet_bootstrap.cc:492] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c: No bootstrap required, opened a new log
I20260807 02:26:04.847700 25979 tablet_bootstrap.cc:492] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8: No bootstrap required, opened a new log
I20260807 02:26:04.847895 25980 ts_tablet_manager.cc:1397] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260807 02:26:04.847903 25979 ts_tablet_manager.cc:1397] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260807 02:26:04.848016 25982 tablet_bootstrap.cc:654] T 3c162c4c4360481984c072a27f84ce0b P 9c464a1082f04ffbacb5dabecc068688: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:04.848838 25982 log.cc:824] T 3c162c4c4360481984c072a27f84ce0b P 9c464a1082f04ffbacb5dabecc068688: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:04.850782 25982 tablet_bootstrap.cc:492] T 3c162c4c4360481984c072a27f84ce0b P 9c464a1082f04ffbacb5dabecc068688: No bootstrap required, opened a new log
I20260807 02:26:04.850986 25982 ts_tablet_manager.cc:1397] T 3c162c4c4360481984c072a27f84ce0b P 9c464a1082f04ffbacb5dabecc068688: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260807 02:26:04.852412 25979 raft_consensus.cc:348] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4343fe518b34c46a90d7dbeece7f96c" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 33299 } } peers { permanent_uuid: "f0c063e6e5d14f8195909de92231e4f8" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33525 } } peers { permanent_uuid: "9c464a1082f04ffbacb5dabecc068688" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33485 } }
I20260807 02:26:04.852412 25980 raft_consensus.cc:348] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4343fe518b34c46a90d7dbeece7f96c" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 33299 } } peers { permanent_uuid: "f0c063e6e5d14f8195909de92231e4f8" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33525 } } peers { permanent_uuid: "9c464a1082f04ffbacb5dabecc068688" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33485 } }
I20260807 02:26:04.852689 25980 raft_consensus.cc:374] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:04.852689 25979 raft_consensus.cc:374] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:04.852751 25979 raft_consensus.cc:729] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0c063e6e5d14f8195909de92231e4f8, State: Initialized, Role: FOLLOWER
I20260807 02:26:04.852759 25980 raft_consensus.cc:729] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4343fe518b34c46a90d7dbeece7f96c, State: Initialized, Role: FOLLOWER
I20260807 02:26:04.853222 25980 consensus_queue.cc:260] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4343fe518b34c46a90d7dbeece7f96c" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 33299 } } peers { permanent_uuid: "f0c063e6e5d14f8195909de92231e4f8" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33525 } } peers { permanent_uuid: "9c464a1082f04ffbacb5dabecc068688" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33485 } }
I20260807 02:26:04.853222 25979 consensus_queue.cc:260] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4343fe518b34c46a90d7dbeece7f96c" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 33299 } } peers { permanent_uuid: "f0c063e6e5d14f8195909de92231e4f8" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33525 } } peers { permanent_uuid: "9c464a1082f04ffbacb5dabecc068688" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33485 } }
I20260807 02:26:04.854176 25823 heartbeater.cc:503] Master 127.4.221.126:44985 was elected leader, sending a full tablet report...
I20260807 02:26:04.854194 25960 heartbeater.cc:503] Master 127.4.221.126:44985 was elected leader, sending a full tablet report...
I20260807 02:26:04.854278 25980 ts_tablet_manager.cc:1428] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260807 02:26:04.854857 25979 ts_tablet_manager.cc:1428] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260807 02:26:04.854874 25982 raft_consensus.cc:348] T 3c162c4c4360481984c072a27f84ce0b P 9c464a1082f04ffbacb5dabecc068688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4343fe518b34c46a90d7dbeece7f96c" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 33299 } } peers { permanent_uuid: "f0c063e6e5d14f8195909de92231e4f8" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33525 } } peers { permanent_uuid: "9c464a1082f04ffbacb5dabecc068688" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33485 } }
I20260807 02:26:04.855127 25982 raft_consensus.cc:374] T 3c162c4c4360481984c072a27f84ce0b P 9c464a1082f04ffbacb5dabecc068688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:04.855196 25982 raft_consensus.cc:729] T 3c162c4c4360481984c072a27f84ce0b P 9c464a1082f04ffbacb5dabecc068688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c464a1082f04ffbacb5dabecc068688, State: Initialized, Role: FOLLOWER
I20260807 02:26:04.855607 25982 consensus_queue.cc:260] T 3c162c4c4360481984c072a27f84ce0b P 9c464a1082f04ffbacb5dabecc068688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4343fe518b34c46a90d7dbeece7f96c" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 33299 } } peers { permanent_uuid: "f0c063e6e5d14f8195909de92231e4f8" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33525 } } peers { permanent_uuid: "9c464a1082f04ffbacb5dabecc068688" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33485 } }
I20260807 02:26:04.856596 25682 heartbeater.cc:503] Master 127.4.221.126:44985 was elected leader, sending a full tablet report...
I20260807 02:26:04.856768 25982 ts_tablet_manager.cc:1428] T 3c162c4c4360481984c072a27f84ce0b P 9c464a1082f04ffbacb5dabecc068688: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
W20260807 02:26:04.903681 25683 tablet.cc:2369] T 3c162c4c4360481984c072a27f84ce0b P 9c464a1082f04ffbacb5dabecc068688: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260807 02:26:04.941905 25986 raft_consensus.cc:482] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260807 02:26:04.942190 25986 raft_consensus.cc:504] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4343fe518b34c46a90d7dbeece7f96c" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 33299 } } peers { permanent_uuid: "f0c063e6e5d14f8195909de92231e4f8" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33525 } } peers { permanent_uuid: "9c464a1082f04ffbacb5dabecc068688" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33485 } }
I20260807 02:26:04.943382 25986 leader_election.cc:288] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4343fe518b34c46a90d7dbeece7f96c (127.4.221.67:33299), 9c464a1082f04ffbacb5dabecc068688 (127.4.221.65:33485)
I20260807 02:26:04.947165 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.943654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.67:33299 (local address 127.4.221.66:41145)
0807 02:26:04.943859 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:04.943874 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:04.943962 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:04.944263 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:04.944266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:04.944283 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:26:04.944406 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.944413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.945139 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.945143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.946027 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.946037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.946693 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.946697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.946855 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.946873 (+ 18us) client_negotiation.cc:769] Sending connection context
0807 02:26:04.946960 (+ 87us) client_negotiation.cc:240] Negotiation successful
0807 02:26:04.947026 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":37}
I20260807 02:26:04.947165 25973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.943768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:41145 (local address 127.4.221.67:33299)
0807 02:26:04.943887 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:04.943893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:04.943926 (+ 33us) server_negotiation.cc:407] Connection header received
0807 02:26:04.944103 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:04.944106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:04.944156 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:04.944245 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:26:04.944524 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.945022 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.946163 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.946555 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.946701 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.946752 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:04.946977 (+ 225us) server_negotiation.cc:299] Negotiation successful
0807 02:26:04.947044 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260807 02:26:04.947255 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.943766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.221.66:56689 (local address 127.4.221.65:33485)
0807 02:26:04.943920 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:04.943925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:04.944162 (+ 237us) server_negotiation.cc:407] Connection header received
0807 02:26:04.944302 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:04.944305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:04.944346 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:04.944425 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0807 02:26:04.944689 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.945158 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.946197 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.946569 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.946698 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.946750 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:04.947048 (+ 298us) server_negotiation.cc:299] Negotiation successful
0807 02:26:04.947127 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260807 02:26:04.947590 25989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.943667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33485 (local address 127.4.221.66:56689)
0807 02:26:04.944058 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:04.944068 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:04.944189 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:04.944437 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:04.944439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:04.944454 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0807 02:26:04.944576 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.944581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.945264 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.945270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.946066 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:04.946075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.946693 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.946697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:04.946874 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.946916 (+ 42us) client_negotiation.cc:769] Sending connection context
0807 02:26:04.947389 (+ 473us) client_negotiation.cc:240] Negotiation successful
0807 02:26:04.947467 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":60,"thread_start_us":108,"threads_started":1}
I20260807 02:26:04.947830 25913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c162c4c4360481984c072a27f84ce0b" candidate_uuid: "f0c063e6e5d14f8195909de92231e4f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4343fe518b34c46a90d7dbeece7f96c" is_pre_election: true
I20260807 02:26:04.948164 25913 raft_consensus.cc:2454] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0c063e6e5d14f8195909de92231e4f8 in term 0.
I20260807 02:26:04.948179 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c162c4c4360481984c072a27f84ce0b" candidate_uuid: "f0c063e6e5d14f8195909de92231e4f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c464a1082f04ffbacb5dabecc068688" is_pre_election: true
I20260807 02:26:04.948421 25634 raft_consensus.cc:2454] T 3c162c4c4360481984c072a27f84ce0b P 9c464a1082f04ffbacb5dabecc068688 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0c063e6e5d14f8195909de92231e4f8 in term 0.
I20260807 02:26:04.948704 25704 leader_election.cc:302] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4343fe518b34c46a90d7dbeece7f96c, f0c063e6e5d14f8195909de92231e4f8; no voters:
I20260807 02:26:04.948999 25986 raft_consensus.cc:2781] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260807 02:26:04.949075 25986 raft_consensus.cc:482] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260807 02:26:04.949105 25986 raft_consensus.cc:3037] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:04.950279 25986 raft_consensus.cc:504] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4343fe518b34c46a90d7dbeece7f96c" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 33299 } } peers { permanent_uuid: "f0c063e6e5d14f8195909de92231e4f8" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33525 } } peers { permanent_uuid: "9c464a1082f04ffbacb5dabecc068688" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33485 } }
I20260807 02:26:04.950626 25986 leader_election.cc:288] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [CANDIDATE]: Term 1 election: Requested vote from peers e4343fe518b34c46a90d7dbeece7f96c (127.4.221.67:33299), 9c464a1082f04ffbacb5dabecc068688 (127.4.221.65:33485)
I20260807 02:26:04.950973 25634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c162c4c4360481984c072a27f84ce0b" candidate_uuid: "f0c063e6e5d14f8195909de92231e4f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c464a1082f04ffbacb5dabecc068688"
I20260807 02:26:04.950990 25913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c162c4c4360481984c072a27f84ce0b" candidate_uuid: "f0c063e6e5d14f8195909de92231e4f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4343fe518b34c46a90d7dbeece7f96c"
I20260807 02:26:04.951081 25634 raft_consensus.cc:3037] T 3c162c4c4360481984c072a27f84ce0b P 9c464a1082f04ffbacb5dabecc068688 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:04.951128 25913 raft_consensus.cc:3037] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:04.952312 25634 raft_consensus.cc:2454] T 3c162c4c4360481984c072a27f84ce0b P 9c464a1082f04ffbacb5dabecc068688 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0c063e6e5d14f8195909de92231e4f8 in term 1.
I20260807 02:26:04.952317 25913 raft_consensus.cc:2454] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0c063e6e5d14f8195909de92231e4f8 in term 1.
I20260807 02:26:04.952661 25703 leader_election.cc:302] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c464a1082f04ffbacb5dabecc068688, f0c063e6e5d14f8195909de92231e4f8; no voters:
I20260807 02:26:04.952844 25986 raft_consensus.cc:2781] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:04.953049 25986 raft_consensus.cc:686] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [term 1 LEADER]: Becoming Leader. State: Replica: f0c063e6e5d14f8195909de92231e4f8, State: Running, Role: LEADER
I20260807 02:26:04.953420 25986 consensus_queue.cc:237] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4343fe518b34c46a90d7dbeece7f96c" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 33299 } } peers { permanent_uuid: "f0c063e6e5d14f8195909de92231e4f8" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33525 } } peers { permanent_uuid: "9c464a1082f04ffbacb5dabecc068688" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33485 } }
I20260807 02:26:04.956619 25480 catalog_manager.cc:5495] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0c063e6e5d14f8195909de92231e4f8 (127.4.221.66). New cstate: current_term: 1 leader_uuid: "f0c063e6e5d14f8195909de92231e4f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4343fe518b34c46a90d7dbeece7f96c" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 33299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0c063e6e5d14f8195909de92231e4f8" member_type: VOTER last_known_addr { host: "127.4.221.66" port: 33525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c464a1082f04ffbacb5dabecc068688" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33485 } health_report { overall_health: UNKNOWN } } }
I20260807 02:26:04.985908 25978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:04.982521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58516 (local address 127.4.221.66:33525)
0807 02:26:04.982638 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:04.982641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:04.982654 (+ 13us) server_negotiation.cc:407] Connection header received
0807 02:26:04.982727 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:04.982729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:04.982784 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:04.982870 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:26:04.983087 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.983592 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.984418 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:04.984643 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:04.984734 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:04.984997 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:26:04.985431 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:26:04.985524 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:04.985634 (+ 110us) server_negotiation.cc:299] Negotiation successful
0807 02:26:04.985681 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 02:26:04.991526 25913 raft_consensus.cc:1261] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c [term 1 FOLLOWER]: Refusing update from remote peer f0c063e6e5d14f8195909de92231e4f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:26:04.991528 25634 raft_consensus.cc:1261] T 3c162c4c4360481984c072a27f84ce0b P 9c464a1082f04ffbacb5dabecc068688 [term 1 FOLLOWER]: Refusing update from remote peer f0c063e6e5d14f8195909de92231e4f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:26:04.992251 25986 consensus_queue.cc:1035] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c464a1082f04ffbacb5dabecc068688" member_type: VOTER last_known_addr { host: "127.4.221.65" port: 33485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:26:04.992452 25990 consensus_queue.cc:1035] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4343fe518b34c46a90d7dbeece7f96c" member_type: VOTER last_known_addr { host: "127.4.221.67" port: 33299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:26:05.001068 25995 mvcc.cc:205] Tried to move back new op lower bound from 7315740938198376448 to 7315740938052165632. Current Snapshot: MvccSnapshot[applied={T|T < 7315740938198376448}]
I20260807 02:26:05.008882 25481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:26:05.009068 25481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260807 02:26:05.011523 25481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e221f758bbe24c0e93efd020be9d9a0b: Sending DeleteTablet for 3 replicas of tablet 3c162c4c4360481984c072a27f84ce0b
I20260807 02:26:05.012508 25891 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c162c4c4360481984c072a27f84ce0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:26:05 UTC) from {username='slave'} at 127.0.0.1:37868
I20260807 02:26:05.012676 25752 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c162c4c4360481984c072a27f84ce0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-07 02:26:05 UTC) from {username='slave'} at 127.0.0.1:58510
I20260807 02:26:05.012953 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 25546
I20260807 02:26:05.016026 26004 tablet_replica.cc:330] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8: stopping tablet replica
I20260807 02:26:05.016415 26004 raft_consensus.cc:2229] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [term 1 LEADER]: Raft consensus shutting down.
I20260807 02:26:05.016861 26004 raft_consensus.cc:2258] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:26:05.018534 26004 ts_tablet_manager.cc:1905] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:26:05.021395 26004 ts_tablet_manager.cc:1918] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:26:05.021497 26004 log.cc:1196] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-1/wal/wals/3c162c4c4360481984c072a27f84ce0b
W20260807 02:26:05.021520 25466 connection.cc:535] client connection to 127.4.221.65:33485 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260807 02:26:05.021692 25466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260807 02:26:05.021852 26004 ts_tablet_manager.cc:1939] T 3c162c4c4360481984c072a27f84ce0b P f0c063e6e5d14f8195909de92231e4f8: Deleting consensus metadata
I20260807 02:26:05.022032 26003 tablet_replica.cc:330] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c: stopping tablet replica
I20260807 02:26:05.022452 26003 raft_consensus.cc:2229] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260807 02:26:05.022512 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 25685
I20260807 02:26:05.022872 26003 raft_consensus.cc:2258] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260807 02:26:05.024003 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:05.023664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.65:33485 (local address 127.0.0.1:44688)
0807 02:26:05.023865 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:05.023927 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.65:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260807 02:26:05.024205 25466 catalog_manager.cc:4584] TS 9c464a1082f04ffbacb5dabecc068688 (127.4.221.65:33485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c162c4c4360481984c072a27f84ce0b: Network error: Client connection negotiation failed: client connection to 127.4.221.65:33485: connect: Connection refused (error 111)
I20260807 02:26:05.025194 26003 ts_tablet_manager.cc:1905] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260807 02:26:05.027407 26003 ts_tablet_manager.cc:1918] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260807 02:26:05.027525 26003 log.cc:1196] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c: Deleting WAL directory at /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786069301823597-4981-0/minicluster-data/ts-2/wal/wals/3c162c4c4360481984c072a27f84ce0b
I20260807 02:26:05.027837 26003 ts_tablet_manager.cc:1939] T 3c162c4c4360481984c072a27f84ce0b P e4343fe518b34c46a90d7dbeece7f96c: Deleting consensus metadata
I20260807 02:26:05.028960 25467 catalog_manager.cc:4841] TS e4343fe518b34c46a90d7dbeece7f96c (127.4.221.67:33299): tablet 3c162c4c4360481984c072a27f84ce0b (table test-table [id=ccd03d5db35a427588f32da5c257e829]) successfully deleted
I20260807 02:26:05.029945 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 25827
I20260807 02:26:05.030459 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:05.030073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.66:33525 (local address 127.0.0.1:58520)
0807 02:26:05.030305 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:05.030365 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.66:33525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260807 02:26:05.037339 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 25445
2026-08-07T02:26:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1097 ms)
[----------] 6 tests from EncryptionPolicyTest (11397 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:05 dist-test-slave-69rg krb5kdc[26009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:05 dist-test-slave-69rg krb5kdc[26009](info): set up 2 sockets
Aug 07 02:26:05 dist-test-slave-69rg krb5kdc[26009](info): commencing operation
krb5kdc: starting...
W20260807 02:26:06.100870 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:06 dist-test-slave-69rg krb5kdc[26009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 02:26:06.234488 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:06.369303 26022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:06.369707 26022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:06.369756 26022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:06.374565 26022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:06.374686 26022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:06.374711 26022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:06.374730 26022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:06.374747 26022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:06.381237 26022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45895
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:06.382977 26022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:06.384372 26022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:06.395217 26030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:06.395217 26028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:06.395217 26027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:06.395525 26022 server_base.cc:1034] running on GCE node
I20260807 02:26:06.396517 26022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:06.398247 26022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:06.399473 26022 hybrid_clock.cc:648] HybridClock initialized: now 1786069566399402 us; error 56 us; skew 500 ppm
Aug 07 02:26:06 dist-test-slave-69rg krb5kdc[26009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069566, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:06.403242 26022 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 02:26:06.404438 26022 webserver.cc:460] Webserver started at http://127.0.0.1:39485/ using document root <none> and password file <none>
I20260807 02:26:06.405352 26022 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:06.405436 26022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:06.405694 26022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:06.408541 26022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "d5f6df824cb544588a4ac254019c3b4f"
format_stamp: "Formatted at 2026-08-07 02:26:06 on dist-test-slave-69rg"
server_key: "6ea027a2d0f4eb7dd3ca5448aa2eb17a"
server_key_iv: "3b4aceacba0050d101f709bf251f83b8"
server_key_version: "encryptionkey@0"
I20260807 02:26:06.409242 26022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "d5f6df824cb544588a4ac254019c3b4f"
format_stamp: "Formatted at 2026-08-07 02:26:06 on dist-test-slave-69rg"
server_key: "6ea027a2d0f4eb7dd3ca5448aa2eb17a"
server_key_iv: "3b4aceacba0050d101f709bf251f83b8"
server_key_version: "encryptionkey@0"
I20260807 02:26:06.414413 26022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260807 02:26:06.417754 26037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:06.419579 26022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:26:06.419782 26022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "d5f6df824cb544588a4ac254019c3b4f"
format_stamp: "Formatted at 2026-08-07 02:26:06 on dist-test-slave-69rg"
server_key: "6ea027a2d0f4eb7dd3ca5448aa2eb17a"
server_key_iv: "3b4aceacba0050d101f709bf251f83b8"
server_key_version: "encryptionkey@0"
I20260807 02:26:06.419965 26022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:06.444244 26022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:06.448568 26022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:06.448853 26022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:06.458333 26022 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45895
I20260807 02:26:06.458366 26092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45895 every 8 connection(s)
I20260807 02:26:06.459654 26022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:06.463567 26093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:06.464370 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 26022
I20260807 02:26:06.464646 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:06.465029 4981 external_mini_cluster.cc:1428] Setting key 448a0d88fadec157f9e07e6280049b50
I20260807 02:26:06.471735 26093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f: Bootstrap starting.
I20260807 02:26:06.475404 26093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f: Neither blocks nor log segments found. Creating new log.
Aug 07 02:26:06 dist-test-slave-69rg krb5kdc[26009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069566, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 02:26:06.476361 26093 log.cc:824] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:06.479064 26093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f: No bootstrap required, opened a new log
I20260807 02:26:06.482059 26096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:06.466983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41039 (local address 127.0.0.1:45895)
0807 02:26:06.467536 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:06.467554 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:06.467597 (+ 43us) server_negotiation.cc:407] Connection header received
0807 02:26:06.468349 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:06.468370 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:06.468795 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:06.469139 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:06.470212 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:06.471342 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:06.472519 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:06.473045 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:06.473173 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:06.476604 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:06.476610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:06.476627 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:06.476660 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:06.478647 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:06.479238 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:06.479244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:06.479249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:06.479369 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:06.479749 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:06.479752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:06.479753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:06.480215 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:06.480469 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:06.480818 (+ 349us) server_negotiation.cc:299] Negotiation successful
0807 02:26:06.481081 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":180,"threads_started":1}
I20260807 02:26:06.483429 26093 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f6df824cb544588a4ac254019c3b4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260807 02:26:06.483713 26093 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:06.483798 26093 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5f6df824cb544588a4ac254019c3b4f, State: Initialized, Role: FOLLOWER
I20260807 02:26:06.484331 26093 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f6df824cb544588a4ac254019c3b4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260807 02:26:06.484514 26093 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:06.484606 26093 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:06.484722 26093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:06.485545 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:06.486110 26093 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f6df824cb544588a4ac254019c3b4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260807 02:26:06.486512 26093 leader_election.cc:302] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5f6df824cb544588a4ac254019c3b4f; no voters:
I20260807 02:26:06.486927 26093 leader_election.cc:288] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:06.487010 26098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:06.487290 26098 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [term 1 LEADER]: Becoming Leader. State: Replica: d5f6df824cb544588a4ac254019c3b4f, State: Running, Role: LEADER
I20260807 02:26:06.487705 26098 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f6df824cb544588a4ac254019c3b4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } }
I20260807 02:26:06.488574 26093 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:06.489133 26098 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5f6df824cb544588a4ac254019c3b4f. Latest consensus state: current_term: 1 leader_uuid: "d5f6df824cb544588a4ac254019c3b4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f6df824cb544588a4ac254019c3b4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } } }
I20260807 02:26:06.489266 26098 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:06.489503 26098 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5f6df824cb544588a4ac254019c3b4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5f6df824cb544588a4ac254019c3b4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45895 } } }
I20260807 02:26:06.489614 26098 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:06.489850 26108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:06.493902 26108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:06.498575 26096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:06.488115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37219 (local address 127.0.0.1:45895)
0807 02:26:06.488248 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:06.488253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:06.488513 (+ 260us) server_negotiation.cc:407] Connection header received
0807 02:26:06.488681 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:06.488684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:06.488743 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:06.488826 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:06.489705 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:06.490526 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:06.491723 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:06.492059 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:06.492134 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:06.494689 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:06.494693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:06.494696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:06.494725 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:06.496581 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:06.497087 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:06.497091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:06.497093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:06.497188 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:06.497555 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:06.497559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:06.497560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:06.497766 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:06.498012 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:06.498289 (+ 277us) server_negotiation.cc:299] Negotiation successful
0807 02:26:06.498402 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 02:26:06.502394 26108 catalog_manager.cc:1347] Generated new cluster ID: 85e91b49afe54db09dec5c31583d2ad5
I20260807 02:26:06.502502 26108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:06.517462 26108 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:06.518734 26108 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:06.533738 26108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5f6df824cb544588a4ac254019c3b4f: Generated new TSK 0
I20260807 02:26:06.534790 26108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:06.542681 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 26022
W20260807 02:26:06.551232 26007 connection.cc:535] client connection to 127.0.0.1:45895 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45895): Connection reset by peer (error 104)
2026-08-07T02:26:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1506 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:06 dist-test-slave-69rg krb5kdc[26128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:06 dist-test-slave-69rg krb5kdc[26128](info): set up 2 sockets
Aug 07 02:26:06 dist-test-slave-69rg krb5kdc[26128](info): commencing operation
krb5kdc: starting...
W20260807 02:26:07.605159 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:07 dist-test-slave-69rg krb5kdc[26128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 02:26:07.736603 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44471
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:07.857321 26142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:07.857625 26142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:07.857698 26142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:07.861882 26142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:07.861967 26142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:07.862015 26142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:07.862069 26142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:07.862106 26142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:07.867375 26142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44471
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:07.868527 26142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:07.869586 26142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:07.878325 26148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:07.878387 26150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:07.878451 26142 server_base.cc:1034] running on GCE node
W20260807 02:26:07.878718 26147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:07.879320 26142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:07.880456 26142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:07.881654 26142 hybrid_clock.cc:648] HybridClock initialized: now 1786069567881638 us; error 25 us; skew 500 ppm
Aug 07 02:26:07 dist-test-slave-69rg krb5kdc[26128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:07.884661 26142 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 02:26:07.885624 26142 webserver.cc:460] Webserver started at http://127.0.0.1:43033/ using document root <none> and password file <none>
I20260807 02:26:07.886273 26142 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:07.886364 26142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:07.886579 26142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:07.888715 26142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "1074f41557a3438fb399282ba25acfff"
format_stamp: "Formatted at 2026-08-07 02:26:07 on dist-test-slave-69rg"
server_key: "233450ad42536199003dafa89d512ff5"
server_key_iv: "cc59584d990fc405b1a6ae8e14538eaf"
server_key_version: "encryptionkey@0"
I20260807 02:26:07.889304 26142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "1074f41557a3438fb399282ba25acfff"
format_stamp: "Formatted at 2026-08-07 02:26:07 on dist-test-slave-69rg"
server_key: "233450ad42536199003dafa89d512ff5"
server_key_iv: "cc59584d990fc405b1a6ae8e14538eaf"
server_key_version: "encryptionkey@0"
I20260807 02:26:07.893327 26142 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260807 02:26:07.896207 26157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:07.897490 26142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:26:07.897652 26142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "1074f41557a3438fb399282ba25acfff"
format_stamp: "Formatted at 2026-08-07 02:26:07 on dist-test-slave-69rg"
server_key: "233450ad42536199003dafa89d512ff5"
server_key_iv: "cc59584d990fc405b1a6ae8e14538eaf"
server_key_version: "encryptionkey@0"
I20260807 02:26:07.897782 26142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:07.910151 26142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:07.913698 26142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:07.913949 26142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:07.921939 26142 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44471
I20260807 02:26:07.921965 26211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44471 every 8 connection(s)
I20260807 02:26:07.922945 26142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:07.923882 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 26142
I20260807 02:26:07.924057 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:07.924422 4981 external_mini_cluster.cc:1428] Setting key 091e7a8768794bb32a178582b77b05df
I20260807 02:26:07.926249 26212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:07.933043 26212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff: Bootstrap starting.
I20260807 02:26:07.934949 26216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:07.925973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37499 (local address 127.0.0.1:44471)
0807 02:26:07.926535 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:07.926554 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:07.926596 (+ 42us) server_negotiation.cc:407] Connection header received
0807 02:26:07.927433 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:07.927454 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:07.927868 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:07.928234 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:07.928627 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:07.930175 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:07.931073 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:07.931759 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:07.931933 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:07.932937 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:07.932943 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:07.932958 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:07.933443 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:07.933507 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:07.933536 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:07.933670 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:07.933880 (+ 210us) server_negotiation.cc:299] Negotiation successful
0807 02:26:07.934007 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":211,"threads_started":1}
I20260807 02:26:07.935731 26212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:07.936645 26212 log.cc:824] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:07.938135 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:07.938684 26212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff: No bootstrap required, opened a new log
I20260807 02:26:07.942831 26212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1074f41557a3438fb399282ba25acfff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44471 } }
I20260807 02:26:07.943094 26212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:07.943145 26212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1074f41557a3438fb399282ba25acfff, State: Initialized, Role: FOLLOWER
I20260807 02:26:07.943611 26212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1074f41557a3438fb399282ba25acfff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44471 } }
I20260807 02:26:07.943747 26212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:07.943840 26212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:07.943943 26212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:07.944839 26216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:07.940545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49711 (local address 127.0.0.1:44471)
0807 02:26:07.940673 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:07.940680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:07.940803 (+ 123us) server_negotiation.cc:407] Connection header received
0807 02:26:07.940897 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:07.940902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:07.940944 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:07.941029 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:07.941320 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:07.942100 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:07.943006 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:07.943353 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:07.943466 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:07.944409 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:07.944413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:07.944416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:07.944428 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:07.944454 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:07.944466 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:07.944562 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:07.944669 (+ 107us) server_negotiation.cc:299] Negotiation successful
0807 02:26:07.944705 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 02:26:07.945124 26212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1074f41557a3438fb399282ba25acfff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44471 } }
I20260807 02:26:07.945484 26212 leader_election.cc:302] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1074f41557a3438fb399282ba25acfff; no voters:
I20260807 02:26:07.945868 26212 leader_election.cc:288] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:07.946033 26223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:07.946346 26223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [term 1 LEADER]: Becoming Leader. State: Replica: 1074f41557a3438fb399282ba25acfff, State: Running, Role: LEADER
I20260807 02:26:07.946691 26223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1074f41557a3438fb399282ba25acfff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44471 } }
I20260807 02:26:07.947163 26212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:07.948396 26224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1074f41557a3438fb399282ba25acfff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1074f41557a3438fb399282ba25acfff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44471 } } }
I20260807 02:26:07.948539 26224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:07.948940 26225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1074f41557a3438fb399282ba25acfff. Latest consensus state: current_term: 1 leader_uuid: "1074f41557a3438fb399282ba25acfff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1074f41557a3438fb399282ba25acfff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44471 } } }
I20260807 02:26:07.949047 26225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff [sys.catalog]: This master's current role is: LEADER
W20260807 02:26:07.952245 26239 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:26:07.952344 26239 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:26:07.952432 26233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:07.953205 26233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:07.958894 26233 catalog_manager.cc:1347] Generated new cluster ID: 5a65a1a863c44bccb031ae0cfd856d93
I20260807 02:26:07.958962 26233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:07.980974 26233 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:07.982102 26233 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:07.994935 26233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1074f41557a3438fb399282ba25acfff: Generated new TSK 0
I20260807 02:26:07.995864 26233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:08.012041 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 26142
W20260807 02:26:08.018841 26126 connection.cc:535] client connection to 127.0.0.1:44471 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44471): Connection reset by peer (error 104)
2026-08-07T02:26:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1467 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:08 dist-test-slave-69rg krb5kdc[26245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:08 dist-test-slave-69rg krb5kdc[26245](info): set up 2 sockets
Aug 07 02:26:08 dist-test-slave-69rg krb5kdc[26245](info): commencing operation
krb5kdc: starting...
W20260807 02:26:09.072872 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:09 dist-test-slave-69rg krb5kdc[26245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 02:26:09.200775 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42919
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:09.311631 26259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:09.311952 26259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:09.312031 26259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:09.316085 26259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:09.316195 26259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:09.316237 26259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:09.316296 26259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:09.316337 26259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:09.321561 26259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42919
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:09.322628 26259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:09.323630 26259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:09.332216 26267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:09.332150 26264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:09.332208 26259 server_base.cc:1034] running on GCE node
W20260807 02:26:09.332149 26265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:09.333067 26259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:09.334299 26259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:09.335513 26259 hybrid_clock.cc:648] HybridClock initialized: now 1786069569335483 us; error 43 us; skew 500 ppm
Aug 07 02:26:09 dist-test-slave-69rg krb5kdc[26245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:09.342691 26259 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 02:26:09.343751 26259 webserver.cc:460] Webserver started at http://127.0.0.1:40781/ using document root <none> and password file <none>
I20260807 02:26:09.344347 26259 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:09.344447 26259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:09.344693 26259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:09.346910 26259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "33935bba7cb845f586d8af49cc1b1226"
format_stamp: "Formatted at 2026-08-07 02:26:09 on dist-test-slave-69rg"
server_key: "07fa9018432fc744f4bbfdf2da2f39cb"
server_key_iv: "4fc33b1ccb0e09ce0135e87b1399d7ac"
server_key_version: "encryptionkey@0"
I20260807 02:26:09.347515 26259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "33935bba7cb845f586d8af49cc1b1226"
format_stamp: "Formatted at 2026-08-07 02:26:09 on dist-test-slave-69rg"
server_key: "07fa9018432fc744f4bbfdf2da2f39cb"
server_key_iv: "4fc33b1ccb0e09ce0135e87b1399d7ac"
server_key_version: "encryptionkey@0"
I20260807 02:26:09.351212 26259 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:26:09.353647 26274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:09.354856 26259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:26:09.355002 26259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "33935bba7cb845f586d8af49cc1b1226"
format_stamp: "Formatted at 2026-08-07 02:26:09 on dist-test-slave-69rg"
server_key: "07fa9018432fc744f4bbfdf2da2f39cb"
server_key_iv: "4fc33b1ccb0e09ce0135e87b1399d7ac"
server_key_version: "encryptionkey@0"
I20260807 02:26:09.355118 26259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:09.359674 26259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:09.362596 26259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:09.362795 26259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:09.370046 26259 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42919
I20260807 02:26:09.370069 26329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42919 every 8 connection(s)
I20260807 02:26:09.370924 26259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:09.373484 26330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:09.377691 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 26259
I20260807 02:26:09.377892 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:09.378229 4981 external_mini_cluster.cc:1428] Setting key 2dd0ba326905ed6ede91d7d8f00513e1
I20260807 02:26:09.379076 26330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226: Bootstrap starting.
I20260807 02:26:09.381189 26330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:09.382165 26330 log.cc:824] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:09.383507 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:09.379654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37449 (local address 127.0.0.1:42919)
0807 02:26:09.380077 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:09.380091 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:09.380167 (+ 76us) server_negotiation.cc:407] Connection header received
0807 02:26:09.380772 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:09.380787 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:09.381043 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:09.381364 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:09.382238 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:09.382266 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:09.382286 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:09.382302 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:09.382367 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:09.382388 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:09.382473 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:09.382647 (+ 174us) server_negotiation.cc:299] Negotiation successful
0807 02:26:09.382799 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":137,"threads_started":1}
I20260807 02:26:09.384184 26330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226: No bootstrap required, opened a new log
I20260807 02:26:09.386377 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:09.388419 26330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33935bba7cb845f586d8af49cc1b1226" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42919 } }
I20260807 02:26:09.388712 26330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:09.388773 26330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33935bba7cb845f586d8af49cc1b1226, State: Initialized, Role: FOLLOWER
I20260807 02:26:09.389212 26330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33935bba7cb845f586d8af49cc1b1226" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42919 } }
I20260807 02:26:09.389376 26330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:09.389467 26330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:09.389570 26330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:09.390146 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:09.388448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42173 (local address 127.0.0.1:42919)
0807 02:26:09.388560 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:09.388563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:09.388789 (+ 226us) server_negotiation.cc:407] Connection header received
0807 02:26:09.388931 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:09.388934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:09.388970 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:09.389044 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:09.389758 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:09.389764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:09.389770 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:09.389792 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:09.389839 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:09.389853 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:09.389948 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:09.390001 (+ 53us) server_negotiation.cc:299] Negotiation successful
0807 02:26:09.390034 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 02:26:09.390728 26330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33935bba7cb845f586d8af49cc1b1226" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42919 } }
I20260807 02:26:09.391081 26330 leader_election.cc:302] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33935bba7cb845f586d8af49cc1b1226; no voters:
I20260807 02:26:09.391378 26330 leader_election.cc:288] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:09.391605 26340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:09.391863 26340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [term 1 LEADER]: Becoming Leader. State: Replica: 33935bba7cb845f586d8af49cc1b1226, State: Running, Role: LEADER
I20260807 02:26:09.392156 26340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33935bba7cb845f586d8af49cc1b1226" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42919 } }
I20260807 02:26:09.392565 26330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:09.393833 26341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33935bba7cb845f586d8af49cc1b1226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33935bba7cb845f586d8af49cc1b1226" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42919 } } }
I20260807 02:26:09.393951 26341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:09.394310 26342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33935bba7cb845f586d8af49cc1b1226. Latest consensus state: current_term: 1 leader_uuid: "33935bba7cb845f586d8af49cc1b1226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33935bba7cb845f586d8af49cc1b1226" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42919 } } }
I20260807 02:26:09.394379 26349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:09.394412 26342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:09.397271 26349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:09.403486 26349 catalog_manager.cc:1347] Generated new cluster ID: 1b1ca6410b21472eace6116d2697ca70
I20260807 02:26:09.403596 26349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:09.433353 26349 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:09.433732 26349 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:09.447134 26349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33935bba7cb845f586d8af49cc1b1226: Generated new TSK 0
I20260807 02:26:09.447813 26349 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:09.456696 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 26259
2026-08-07T02:26:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:09 dist-test-slave-69rg krb5kdc[26362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:09 dist-test-slave-69rg krb5kdc[26362](info): set up 2 sockets
Aug 07 02:26:09 dist-test-slave-69rg krb5kdc[26362](info): commencing operation
krb5kdc: starting...
W20260807 02:26:10.545763 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:10 dist-test-slave-69rg krb5kdc[26362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 02:26:10.678042 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:10.790750 26375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:10.791033 26375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:10.791109 26375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:10.795055 26375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:10.795156 26375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:10.795195 26375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:10.795233 26375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:10.795264 26375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:10.800604 26375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:10.801720 26375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:10.802759 26375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 02:26:10.811714 26375 server_base.cc:1034] running on GCE node
W20260807 02:26:10.811676 26381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:10.811692 26380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:10.811655 26383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:10.812398 26375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:10.813452 26375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:10.814635 26375 hybrid_clock.cc:648] HybridClock initialized: now 1786069570814589 us; error 58 us; skew 500 ppm
Aug 07 02:26:10 dist-test-slave-69rg krb5kdc[26362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:10.817557 26375 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 02:26:10.818604 26375 webserver.cc:460] Webserver started at http://127.0.0.1:43427/ using document root <none> and password file <none>
I20260807 02:26:10.819161 26375 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:10.819227 26375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:10.819418 26375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:10.821534 26375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "78641b1ddadb4bcca399dec343e0b1dc"
format_stamp: "Formatted at 2026-08-07 02:26:10 on dist-test-slave-69rg"
server_key: "428df312331e15851c7be0ac83851b65"
server_key_iv: "19604e03413337a5b83c8be131a8669d"
server_key_version: "encryptionkey@0"
I20260807 02:26:10.822170 26375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "78641b1ddadb4bcca399dec343e0b1dc"
format_stamp: "Formatted at 2026-08-07 02:26:10 on dist-test-slave-69rg"
server_key: "428df312331e15851c7be0ac83851b65"
server_key_iv: "19604e03413337a5b83c8be131a8669d"
server_key_version: "encryptionkey@0"
I20260807 02:26:10.826105 26375 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260807 02:26:10.828486 26391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:10.829746 26375 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:26:10.829917 26375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "78641b1ddadb4bcca399dec343e0b1dc"
format_stamp: "Formatted at 2026-08-07 02:26:10 on dist-test-slave-69rg"
server_key: "428df312331e15851c7be0ac83851b65"
server_key_iv: "19604e03413337a5b83c8be131a8669d"
server_key_version: "encryptionkey@0"
I20260807 02:26:10.830013 26375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:10.847560 26375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:10.850608 26375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:10.850772 26375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:10.858194 26375 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34009
I20260807 02:26:10.858341 26445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34009 every 8 connection(s)
I20260807 02:26:10.859184 26375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:10.862252 26446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:10.865257 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 26375
I20260807 02:26:10.865423 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:10.865804 4981 external_mini_cluster.cc:1428] Setting key 68a7d93819343faf3651ca86a9af314f
I20260807 02:26:10.869065 26446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc: Bootstrap starting.
I20260807 02:26:10.871937 26446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:10.872952 26446 log.cc:824] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:10.875257 26446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc: No bootstrap required, opened a new log
Aug 07 02:26:10 dist-test-slave-69rg krb5kdc[26362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 02:26:10.879817 26446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78641b1ddadb4bcca399dec343e0b1dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260807 02:26:10.880141 26446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:10.880213 26446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78641b1ddadb4bcca399dec343e0b1dc, State: Initialized, Role: FOLLOWER
I20260807 02:26:10.880707 26446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78641b1ddadb4bcca399dec343e0b1dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260807 02:26:10.880874 26446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:10.880985 26446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:10.881114 26446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:10.882215 26446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78641b1ddadb4bcca399dec343e0b1dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260807 02:26:10.882299 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:10.867376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58055 (local address 127.0.0.1:34009)
0807 02:26:10.867926 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:10.867941 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:10.867976 (+ 35us) server_negotiation.cc:407] Connection header received
0807 02:26:10.868608 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:10.868625 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:10.868967 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:10.869317 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:10.870378 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:10.871411 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:10.872788 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:10.873371 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:10.873502 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:10.876355 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:10.876362 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:10.876379 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:10.876412 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:10.879195 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:10.879636 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:10.879642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:10.879648 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:10.879803 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:10.880073 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:10.880076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:10.880078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:10.880610 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:10.880832 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:10.881110 (+ 278us) server_negotiation.cc:299] Negotiation successful
0807 02:26:10.881448 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":214,"threads_started":1}
I20260807 02:26:10.882618 26446 leader_election.cc:302] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78641b1ddadb4bcca399dec343e0b1dc; no voters:
I20260807 02:26:10.882974 26446 leader_election.cc:288] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:10.883093 26451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:10.883354 26451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [term 1 LEADER]: Becoming Leader. State: Replica: 78641b1ddadb4bcca399dec343e0b1dc, State: Running, Role: LEADER
I20260807 02:26:10.883761 26451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78641b1ddadb4bcca399dec343e0b1dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260807 02:26:10.884161 26446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:10.884938 26453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78641b1ddadb4bcca399dec343e0b1dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78641b1ddadb4bcca399dec343e0b1dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } } }
I20260807 02:26:10.885058 26453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:10.885360 26453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78641b1ddadb4bcca399dec343e0b1dc. Latest consensus state: current_term: 1 leader_uuid: "78641b1ddadb4bcca399dec343e0b1dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78641b1ddadb4bcca399dec343e0b1dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } } }
I20260807 02:26:10.885435 26453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:10.885516 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:10.885763 26459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:10.889083 26459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:10.895321 26459 catalog_manager.cc:1347] Generated new cluster ID: 701d4c180e324e0ea4ea4fc82e4e46f4
I20260807 02:26:10.895430 26459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:10.897238 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:10.888091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44899 (local address 127.0.0.1:34009)
0807 02:26:10.888191 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:10.888195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:10.888214 (+ 19us) server_negotiation.cc:407] Connection header received
0807 02:26:10.888467 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:10.888472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:10.888522 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:10.888610 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:10.889433 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:10.890262 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:10.891399 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:10.891670 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:10.891786 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:10.893606 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:10.893609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:10.893612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:10.893637 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:10.895349 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:10.895931 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:10.895935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:10.895937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:10.896031 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:10.896357 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:10.896362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:10.896364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:10.896585 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:10.896719 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:10.896954 (+ 235us) server_negotiation.cc:299] Negotiation successful
0807 02:26:10.897079 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260807 02:26:10.906118 26459 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:10.907348 26459 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:10.915647 26459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78641b1ddadb4bcca399dec343e0b1dc: Generated new TSK 0
I20260807 02:26:10.916510 26459 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:10.927613 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 26375
2026-08-07T02:26:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1470 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:10 dist-test-slave-69rg krb5kdc[26479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:10 dist-test-slave-69rg krb5kdc[26479](info): set up 2 sockets
Aug 07 02:26:10 dist-test-slave-69rg krb5kdc[26479](info): commencing operation
krb5kdc: starting...
W20260807 02:26:12.005268 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:12 dist-test-slave-69rg krb5kdc[26479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 02:26:12.144606 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35493
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:12.281424 26493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:12.281736 26493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:12.281832 26493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:12.285881 26493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:12.285972 26493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:12.286011 26493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:12.286069 26493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:12.286109 26493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:12.291432 26493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:12.292512 26493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:12.293551 26493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:12.302452 26501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:12.302629 26493 server_base.cc:1034] running on GCE node
W20260807 02:26:12.302505 26499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:12.302505 26498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:12.303262 26493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:12.304385 26493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:12.305583 26493 hybrid_clock.cc:648] HybridClock initialized: now 1786069572305562 us; error 38 us; skew 500 ppm
Aug 07 02:26:12 dist-test-slave-69rg krb5kdc[26479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:12.308529 26493 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 02:26:12.309418 26493 webserver.cc:460] Webserver started at http://127.0.0.1:36951/ using document root <none> and password file <none>
I20260807 02:26:12.310078 26493 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:12.310171 26493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:12.310411 26493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:12.312503 26493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "5aac3c17513d45a1b11bc433001d67a6"
format_stamp: "Formatted at 2026-08-07 02:26:12 on dist-test-slave-69rg"
server_key: "d7a648c335fc22445b8ead4a3242bd3e"
server_key_iv: "63e83dae4ad7bae73e2ec6fbd5390e85"
server_key_version: "encryptionkey@0"
I20260807 02:26:12.313115 26493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "5aac3c17513d45a1b11bc433001d67a6"
format_stamp: "Formatted at 2026-08-07 02:26:12 on dist-test-slave-69rg"
server_key: "d7a648c335fc22445b8ead4a3242bd3e"
server_key_iv: "63e83dae4ad7bae73e2ec6fbd5390e85"
server_key_version: "encryptionkey@0"
I20260807 02:26:12.316870 26493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260807 02:26:12.319425 26509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:12.320565 26493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 02:26:12.320713 26493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "5aac3c17513d45a1b11bc433001d67a6"
format_stamp: "Formatted at 2026-08-07 02:26:12 on dist-test-slave-69rg"
server_key: "d7a648c335fc22445b8ead4a3242bd3e"
server_key_iv: "63e83dae4ad7bae73e2ec6fbd5390e85"
server_key_version: "encryptionkey@0"
I20260807 02:26:12.320827 26493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:12.348297 26493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:12.351573 26493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:12.351814 26493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:12.359266 26493 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35493
I20260807 02:26:12.359297 26562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35493 every 8 connection(s)
I20260807 02:26:12.360270 26493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:12.362392 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 26493
I20260807 02:26:12.362553 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:12.362869 4981 external_mini_cluster.cc:1428] Setting key fd8c62e91fd6086e71a4876018689714
I20260807 02:26:12.363358 26564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:12.370052 26564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6: Bootstrap starting.
I20260807 02:26:12.372124 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:12.364415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57247 (local address 127.0.0.1:35493)
0807 02:26:12.364921 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:12.364937 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:12.364975 (+ 38us) server_negotiation.cc:407] Connection header received
0807 02:26:12.365662 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:12.365679 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:12.366100 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:12.366449 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:12.366872 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:12.367874 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:12.368824 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:12.369290 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:12.369525 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:12.370473 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:12.370478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:12.370490 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:12.370883 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:12.370937 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:12.370960 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:12.371083 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:12.371201 (+ 118us) server_negotiation.cc:299] Negotiation successful
0807 02:26:12.371419 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":180,"threads_started":1}
I20260807 02:26:12.372412 26564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:12.373278 26564 log.cc:824] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:12.375106 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:12.375416 26564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6: No bootstrap required, opened a new log
I20260807 02:26:12.379393 26564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aac3c17513d45a1b11bc433001d67a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } }
I20260807 02:26:12.379675 26564 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:12.379724 26564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aac3c17513d45a1b11bc433001d67a6, State: Initialized, Role: FOLLOWER
I20260807 02:26:12.380215 26564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aac3c17513d45a1b11bc433001d67a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } }
I20260807 02:26:12.380374 26564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:12.380466 26564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:12.380558 26564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:12.381690 26564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aac3c17513d45a1b11bc433001d67a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } }
I20260807 02:26:12.382042 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:12.377341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42541 (local address 127.0.0.1:35493)
0807 02:26:12.377476 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:12.377481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:12.377663 (+ 182us) server_negotiation.cc:407] Connection header received
0807 02:26:12.377790 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:12.377792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:12.377896 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:12.377963 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:12.378276 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:12.379128 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:12.380244 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:12.380596 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:12.380710 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:12.381567 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:12.381571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:12.381573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:12.381586 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:12.381616 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:12.381626 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:12.381701 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:12.381860 (+ 159us) server_negotiation.cc:299] Negotiation successful
0807 02:26:12.381916 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260807 02:26:12.382089 26564 leader_election.cc:302] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aac3c17513d45a1b11bc433001d67a6; no voters:
I20260807 02:26:12.382373 26564 leader_election.cc:288] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:12.382503 26574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:12.382735 26574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5aac3c17513d45a1b11bc433001d67a6, State: Running, Role: LEADER
I20260807 02:26:12.383219 26574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aac3c17513d45a1b11bc433001d67a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } }
I20260807 02:26:12.383615 26564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:12.384399 26574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aac3c17513d45a1b11bc433001d67a6. Latest consensus state: current_term: 1 leader_uuid: "5aac3c17513d45a1b11bc433001d67a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aac3c17513d45a1b11bc433001d67a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } } }
I20260807 02:26:12.384532 26574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:12.384859 26575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aac3c17513d45a1b11bc433001d67a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aac3c17513d45a1b11bc433001d67a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35493 } } }
I20260807 02:26:12.384970 26575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6 [sys.catalog]: This master's current role is: LEADER
W20260807 02:26:12.388662 26590 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:26:12.388756 26590 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:26:12.388847 26583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:12.389685 26583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:12.397006 26583 catalog_manager.cc:1347] Generated new cluster ID: 8ab598e046f2439fa7bdefccef59d362
I20260807 02:26:12.397110 26583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:12.434839 26583 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:12.436131 26583 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:12.441642 26583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aac3c17513d45a1b11bc433001d67a6: Generated new TSK 0
I20260807 02:26:12.442341 26583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:12.452683 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 26493
2026-08-07T02:26:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1524 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:12 dist-test-slave-69rg krb5kdc[26596](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:12 dist-test-slave-69rg krb5kdc[26596](info): set up 2 sockets
Aug 07 02:26:12 dist-test-slave-69rg krb5kdc[26596](info): commencing operation
krb5kdc: starting...
W20260807 02:26:13.516881 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:13 dist-test-slave-69rg krb5kdc[26596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 02:26:13.646967 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:13.758787 26610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:13.759070 26610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:13.759112 26610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:13.763296 26610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:13.763360 26610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:13.763379 26610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:13.763435 26610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:13.763468 26610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:13.768836 26610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:13.770021 26610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:13.771081 26610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:13.780241 26615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:13.780232 26616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:13.780228 26618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:13.780366 26610 server_base.cc:1034] running on GCE node
I20260807 02:26:13.781142 26610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:13.782336 26610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:13.783529 26610 hybrid_clock.cc:648] HybridClock initialized: now 1786069573783502 us; error 44 us; skew 500 ppm
Aug 07 02:26:13 dist-test-slave-69rg krb5kdc[26596](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:13.786374 26610 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 02:26:13.787312 26610 webserver.cc:460] Webserver started at http://127.0.0.1:33405/ using document root <none> and password file <none>
I20260807 02:26:13.788059 26610 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:13.788165 26610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:13.788406 26610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:13.791042 26610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "32941040c3124f2ba3cf4fb503ec4b86"
format_stamp: "Formatted at 2026-08-07 02:26:13 on dist-test-slave-69rg"
server_key: "b7018fd8c6e5d92e8f1c11b021a07712"
server_key_iv: "c54592add4aa69206b9619a89d120569"
server_key_version: "encryptionkey@0"
I20260807 02:26:13.791648 26610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "32941040c3124f2ba3cf4fb503ec4b86"
format_stamp: "Formatted at 2026-08-07 02:26:13 on dist-test-slave-69rg"
server_key: "b7018fd8c6e5d92e8f1c11b021a07712"
server_key_iv: "c54592add4aa69206b9619a89d120569"
server_key_version: "encryptionkey@0"
I20260807 02:26:13.795650 26610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260807 02:26:13.798146 26625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:13.799386 26610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 02:26:13.799535 26610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "32941040c3124f2ba3cf4fb503ec4b86"
format_stamp: "Formatted at 2026-08-07 02:26:13 on dist-test-slave-69rg"
server_key: "b7018fd8c6e5d92e8f1c11b021a07712"
server_key_iv: "c54592add4aa69206b9619a89d120569"
server_key_version: "encryptionkey@0"
I20260807 02:26:13.799656 26610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:13.804435 26610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:13.807358 26610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:13.807555 26610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:13.814960 26610 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45643
I20260807 02:26:13.814988 26679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45643 every 8 connection(s)
I20260807 02:26:13.815939 26610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:13.818624 26680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:13.823825 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 26610
I20260807 02:26:13.823988 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:13.824319 4981 external_mini_cluster.cc:1428] Setting key 9d2ba5f2eccff304a5363b9a0b8a5d38
I20260807 02:26:13.824389 26680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86: Bootstrap starting.
I20260807 02:26:13.826967 26680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:13.827692 26680 log.cc:824] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:13.829550 26683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:13.825741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53207 (local address 127.0.0.1:45643)
0807 02:26:13.826234 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:13.826251 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:13.826284 (+ 33us) server_negotiation.cc:407] Connection header received
0807 02:26:13.826919 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:13.826934 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:13.827262 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:13.827592 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:13.828420 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:13.828424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:13.828434 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:13.828443 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:13.828485 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:13.828501 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:13.828622 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:13.828741 (+ 119us) server_negotiation.cc:299] Negotiation successful
0807 02:26:13.828899 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":145,"threads_started":1}
I20260807 02:26:13.829890 26680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86: No bootstrap required, opened a new log
I20260807 02:26:13.832430 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:13.834113 26680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32941040c3124f2ba3cf4fb503ec4b86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45643 } }
I20260807 02:26:13.834378 26680 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:13.834488 26680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32941040c3124f2ba3cf4fb503ec4b86, State: Initialized, Role: FOLLOWER
I20260807 02:26:13.834975 26680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32941040c3124f2ba3cf4fb503ec4b86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45643 } }
I20260807 02:26:13.835153 26680 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:13.835254 26680 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:13.835376 26680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:13.836288 26683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:13.834482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33035 (local address 127.0.0.1:45643)
0807 02:26:13.834608 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:13.834613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:13.834890 (+ 277us) server_negotiation.cc:407] Connection header received
0807 02:26:13.834978 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:13.834981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:13.835021 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:13.835111 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:13.835930 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:13.835934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:13.835936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:13.835950 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:13.835978 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:13.835990 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:13.836083 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:13.836118 (+ 35us) server_negotiation.cc:299] Negotiation successful
0807 02:26:13.836151 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 02:26:13.836555 26680 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32941040c3124f2ba3cf4fb503ec4b86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45643 } }
I20260807 02:26:13.836926 26680 leader_election.cc:302] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32941040c3124f2ba3cf4fb503ec4b86; no voters:
I20260807 02:26:13.837262 26680 leader_election.cc:288] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:13.837395 26690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:13.837642 26690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [term 1 LEADER]: Becoming Leader. State: Replica: 32941040c3124f2ba3cf4fb503ec4b86, State: Running, Role: LEADER
I20260807 02:26:13.838027 26690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32941040c3124f2ba3cf4fb503ec4b86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45643 } }
I20260807 02:26:13.838570 26680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:13.839839 26692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32941040c3124f2ba3cf4fb503ec4b86. Latest consensus state: current_term: 1 leader_uuid: "32941040c3124f2ba3cf4fb503ec4b86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32941040c3124f2ba3cf4fb503ec4b86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45643 } } }
I20260807 02:26:13.839840 26691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32941040c3124f2ba3cf4fb503ec4b86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32941040c3124f2ba3cf4fb503ec4b86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45643 } } }
I20260807 02:26:13.839984 26692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:13.839990 26691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:13.840480 26699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:13.843726 26699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:13.849124 26699 catalog_manager.cc:1347] Generated new cluster ID: 1403e1f2f0f44a14ab5e2fb5d9ccfebd
I20260807 02:26:13.849192 26699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:13.871624 26699 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:13.871996 26699 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:13.880574 26699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32941040c3124f2ba3cf4fb503ec4b86: Generated new TSK 0
I20260807 02:26:13.881289 26699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:13.885531 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 26610
2026-08-07T02:26:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:13 dist-test-slave-69rg krb5kdc[26711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:13 dist-test-slave-69rg krb5kdc[26711](info): set up 2 sockets
Aug 07 02:26:13 dist-test-slave-69rg krb5kdc[26711](info): commencing operation
krb5kdc: starting...
W20260807 02:26:14.953080 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:15 dist-test-slave-69rg krb5kdc[26711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 02:26:15.081166 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34333
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:15.191761 26725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:15.192049 26725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:15.192129 26725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:15.196177 26725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:15.196274 26725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:15.196312 26725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:15.196368 26725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:15.196405 26725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:15.201627 26725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:15.202721 26725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:15.203748 26725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:15.212306 26733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:15.212371 26725 server_base.cc:1034] running on GCE node
W20260807 02:26:15.212338 26730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:15.212602 26731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:15.213208 26725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:15.214397 26725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:15.215585 26725 hybrid_clock.cc:648] HybridClock initialized: now 1786069575215543 us; error 69 us; skew 500 ppm
Aug 07 02:26:15 dist-test-slave-69rg krb5kdc[26711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:15.223529 26725 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 02:26:15.224536 26725 webserver.cc:460] Webserver started at http://127.0.0.1:45661/ using document root <none> and password file <none>
I20260807 02:26:15.225119 26725 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:15.225188 26725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:15.225363 26725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:15.227572 26725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "53236ca7e2a84cf0adec56b8163203fd"
format_stamp: "Formatted at 2026-08-07 02:26:15 on dist-test-slave-69rg"
server_key: "13aba81dfdfcdfa97103574830160345"
server_key_iv: "696727f35e95bae4127eb68ba6d0c434"
server_key_version: "encryptionkey@0"
I20260807 02:26:15.228102 26725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "53236ca7e2a84cf0adec56b8163203fd"
format_stamp: "Formatted at 2026-08-07 02:26:15 on dist-test-slave-69rg"
server_key: "13aba81dfdfcdfa97103574830160345"
server_key_iv: "696727f35e95bae4127eb68ba6d0c434"
server_key_version: "encryptionkey@0"
I20260807 02:26:15.232038 26725 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260807 02:26:15.234463 26742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:15.235706 26725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:26:15.235836 26725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "53236ca7e2a84cf0adec56b8163203fd"
format_stamp: "Formatted at 2026-08-07 02:26:15 on dist-test-slave-69rg"
server_key: "13aba81dfdfcdfa97103574830160345"
server_key_iv: "696727f35e95bae4127eb68ba6d0c434"
server_key_version: "encryptionkey@0"
I20260807 02:26:15.235927 26725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:15.252116 26725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:15.255330 26725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:15.255499 26725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:15.262786 26725 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34333
I20260807 02:26:15.262816 26796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34333 every 8 connection(s)
I20260807 02:26:15.263733 26725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:15.266502 26797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:15.268047 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 26725
I20260807 02:26:15.268208 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:15.268667 4981 external_mini_cluster.cc:1428] Setting key 39818237d7d6f5835b297d621a3c296f
I20260807 02:26:15.272996 26797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd: Bootstrap starting.
I20260807 02:26:15.275492 26797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:15.276445 26797 log.cc:824] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd: Log is configured to *not* fsync() on all Append() calls
Aug 07 02:26:15 dist-test-slave-69rg krb5kdc[26711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 02:26:15.278666 26797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd: No bootstrap required, opened a new log
I20260807 02:26:15.282814 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:15.270115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:60709 (local address 127.0.0.1:34333)
0807 02:26:15.270566 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:15.270581 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:15.270616 (+ 35us) server_negotiation.cc:407] Connection header received
0807 02:26:15.271311 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:15.271330 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:15.271661 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:15.272005 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:15.272817 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:15.273916 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:15.274781 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:15.275252 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:15.275422 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:15.278163 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:15.278168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:15.278183 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:15.278209 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:15.279979 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:15.280503 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:15.280509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:15.280513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:15.280620 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:15.280969 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:15.280972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:15.280974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:15.281343 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:15.281575 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:15.281888 (+ 313us) server_negotiation.cc:299] Negotiation successful
0807 02:26:15.282104 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":140,"threads_started":1}
I20260807 02:26:15.282716 26797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53236ca7e2a84cf0adec56b8163203fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34333 } }
I20260807 02:26:15.282960 26797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:15.283037 26797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53236ca7e2a84cf0adec56b8163203fd, State: Initialized, Role: FOLLOWER
I20260807 02:26:15.283460 26797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53236ca7e2a84cf0adec56b8163203fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34333 } }
I20260807 02:26:15.283588 26797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:15.283680 26797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:15.283831 26797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:15.284922 26797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53236ca7e2a84cf0adec56b8163203fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34333 } }
I20260807 02:26:15.285344 26797 leader_election.cc:302] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53236ca7e2a84cf0adec56b8163203fd; no voters:
I20260807 02:26:15.285707 26797 leader_election.cc:288] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:15.285791 26802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:15.285913 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:15.286062 26802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [term 1 LEADER]: Becoming Leader. State: Replica: 53236ca7e2a84cf0adec56b8163203fd, State: Running, Role: LEADER
I20260807 02:26:15.286341 26802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53236ca7e2a84cf0adec56b8163203fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34333 } }
I20260807 02:26:15.287096 26797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:15.288130 26802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53236ca7e2a84cf0adec56b8163203fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53236ca7e2a84cf0adec56b8163203fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34333 } } }
I20260807 02:26:15.288254 26802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:15.288535 26803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53236ca7e2a84cf0adec56b8163203fd. Latest consensus state: current_term: 1 leader_uuid: "53236ca7e2a84cf0adec56b8163203fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53236ca7e2a84cf0adec56b8163203fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34333 } } }
I20260807 02:26:15.288640 26803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:15.288748 26817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:15.291982 26817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:15.298820 26817 catalog_manager.cc:1347] Generated new cluster ID: 5eb3f915affe46fba85eea7c2aeed617
I20260807 02:26:15.298898 26817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:15.300127 26800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:15.289662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:60851 (local address 127.0.0.1:34333)
0807 02:26:15.289794 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:15.289801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:15.289840 (+ 39us) server_negotiation.cc:407] Connection header received
0807 02:26:15.290037 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:15.290043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:15.290106 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:15.290187 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:15.291227 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:15.292053 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:15.293342 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:15.293615 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:15.293735 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:15.295765 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:15.295769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:15.295772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:15.295802 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:15.297683 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:15.298308 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:15.298312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:15.298314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:15.298433 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:15.298976 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:15.298980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:15.298982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:15.299218 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:15.299359 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:15.299848 (+ 489us) server_negotiation.cc:299] Negotiation successful
0807 02:26:15.299965 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260807 02:26:15.329016 26817 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:15.330222 26817 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:15.338173 26817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53236ca7e2a84cf0adec56b8163203fd: Generated new TSK 0
I20260807 02:26:15.339051 26817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:15.343331 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 26725
W20260807 02:26:15.349977 26709 connection.cc:535] client connection to 127.0.0.1:34333 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34333): Connection reset by peer (error 104)
2026-08-07T02:26:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1457 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:15 dist-test-slave-69rg krb5kdc[26829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:15 dist-test-slave-69rg krb5kdc[26829](info): set up 2 sockets
Aug 07 02:26:15 dist-test-slave-69rg krb5kdc[26829](info): commencing operation
krb5kdc: starting...
W20260807 02:26:16.412534 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:16 dist-test-slave-69rg krb5kdc[26829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 02:26:16.553056 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:16.663465 26842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:16.663748 26842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:16.663791 26842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:16.667835 26842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:16.667905 26842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:16.667925 26842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:16.667944 26842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:16.667959 26842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:16.673374 26842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:16.674528 26842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:16.675566 26842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:16.684363 26847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:16.684372 26850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:16.684381 26848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:16.684967 26842 server_base.cc:1034] running on GCE node
I20260807 02:26:16.685503 26842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:16.686628 26842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:16.687806 26842 hybrid_clock.cc:648] HybridClock initialized: now 1786069576687779 us; error 37 us; skew 500 ppm
Aug 07 02:26:16 dist-test-slave-69rg krb5kdc[26829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:16.690858 26842 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 02:26:16.691748 26842 webserver.cc:460] Webserver started at http://127.0.0.1:36523/ using document root <none> and password file <none>
I20260807 02:26:16.692346 26842 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:16.692443 26842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:16.692680 26842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:16.694805 26842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "c7bc9052cb164352a18c707bc1d4fc0e"
format_stamp: "Formatted at 2026-08-07 02:26:16 on dist-test-slave-69rg"
server_key: "bf2a625a621787c69c6f2fa1c1222b41"
server_key_iv: "e049d6bae2a72f7957523d1b685ddf04"
server_key_version: "encryptionkey@0"
I20260807 02:26:16.695400 26842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "c7bc9052cb164352a18c707bc1d4fc0e"
format_stamp: "Formatted at 2026-08-07 02:26:16 on dist-test-slave-69rg"
server_key: "bf2a625a621787c69c6f2fa1c1222b41"
server_key_iv: "e049d6bae2a72f7957523d1b685ddf04"
server_key_version: "encryptionkey@0"
I20260807 02:26:16.699076 26842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:26:16.701581 26857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:16.702860 26842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:26:16.703004 26842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "c7bc9052cb164352a18c707bc1d4fc0e"
format_stamp: "Formatted at 2026-08-07 02:26:16 on dist-test-slave-69rg"
server_key: "bf2a625a621787c69c6f2fa1c1222b41"
server_key_iv: "e049d6bae2a72f7957523d1b685ddf04"
server_key_version: "encryptionkey@0"
I20260807 02:26:16.703118 26842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:16.725908 26842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:16.729388 26842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:16.729611 26842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:16.737332 26842 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43941
I20260807 02:26:16.737362 26912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43941 every 8 connection(s)
I20260807 02:26:16.738377 26842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:16.740098 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 26842
I20260807 02:26:16.740254 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:16.740556 4981 external_mini_cluster.cc:1428] Setting key 95004870483dadecb645058beb08016b
I20260807 02:26:16.741226 26913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:16.747977 26913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e: Bootstrap starting.
I20260807 02:26:16.749913 26917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:16.742009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:48345 (local address 127.0.0.1:43941)
0807 02:26:16.742428 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:16.742444 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:16.742478 (+ 34us) server_negotiation.cc:407] Connection header received
0807 02:26:16.743138 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:16.743153 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:16.743477 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:16.743756 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:16.744209 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:16.745247 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:16.746473 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:16.746943 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:16.747059 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:16.747992 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:16.747997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:16.748009 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:16.748460 (+ 451us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:16.748514 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:16.748537 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:16.748648 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:16.748926 (+ 278us) server_negotiation.cc:299] Negotiation successful
0807 02:26:16.749076 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":147,"threads_started":1}
I20260807 02:26:16.750492 26913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:16.751304 26913 log.cc:824] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:16.752972 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:16.753458 26913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e: No bootstrap required, opened a new log
I20260807 02:26:16.757761 26913 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7bc9052cb164352a18c707bc1d4fc0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43941 } }
I20260807 02:26:16.758137 26913 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:16.758226 26913 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7bc9052cb164352a18c707bc1d4fc0e, State: Initialized, Role: FOLLOWER
I20260807 02:26:16.758682 26913 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7bc9052cb164352a18c707bc1d4fc0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43941 } }
I20260807 02:26:16.758811 26913 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:16.758927 26913 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:16.759056 26913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:16.759454 26917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:16.755141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:38341 (local address 127.0.0.1:43941)
0807 02:26:16.755283 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:16.755288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:16.755424 (+ 136us) server_negotiation.cc:407] Connection header received
0807 02:26:16.755510 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:16.755514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:16.755573 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:16.755677 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:16.755913 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:16.756667 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:16.757490 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:16.757855 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:16.757962 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:16.758953 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:16.758956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:16.758959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:16.758973 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:16.759006 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:16.759038 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:16.759127 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:16.759263 (+ 136us) server_negotiation.cc:299] Negotiation successful
0807 02:26:16.759307 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:26:16.760341 26913 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7bc9052cb164352a18c707bc1d4fc0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43941 } }
I20260807 02:26:16.760721 26913 leader_election.cc:302] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7bc9052cb164352a18c707bc1d4fc0e; no voters:
I20260807 02:26:16.761088 26913 leader_election.cc:288] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:16.761193 26924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:16.761485 26924 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [term 1 LEADER]: Becoming Leader. State: Replica: c7bc9052cb164352a18c707bc1d4fc0e, State: Running, Role: LEADER
I20260807 02:26:16.761898 26924 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7bc9052cb164352a18c707bc1d4fc0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43941 } }
I20260807 02:26:16.762462 26913 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:16.763573 26925 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7bc9052cb164352a18c707bc1d4fc0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7bc9052cb164352a18c707bc1d4fc0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43941 } } }
I20260807 02:26:16.763747 26925 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:16.764117 26926 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7bc9052cb164352a18c707bc1d4fc0e. Latest consensus state: current_term: 1 leader_uuid: "c7bc9052cb164352a18c707bc1d4fc0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7bc9052cb164352a18c707bc1d4fc0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43941 } } }
I20260807 02:26:16.764178 26933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:16.764248 26926 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:16.767448 26933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:16.773185 26933 catalog_manager.cc:1347] Generated new cluster ID: c168cbb2141040c0a71d6794e8368172
I20260807 02:26:16.773285 26933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:16.786203 26933 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:16.787113 26933 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:16.791251 26933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7bc9052cb164352a18c707bc1d4fc0e: Generated new TSK 0
I20260807 02:26:16.791859 26933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:16.795593 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 26842
W20260807 02:26:16.803179 26827 connection.cc:535] client connection to 127.0.0.1:43941 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43941): Connection reset by peer (error 104)
2026-08-07T02:26:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:16 dist-test-slave-69rg krb5kdc[26945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:16 dist-test-slave-69rg krb5kdc[26945](info): set up 2 sockets
Aug 07 02:26:16 dist-test-slave-69rg krb5kdc[26945](info): commencing operation
krb5kdc: starting...
W20260807 02:26:17.854074 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:17 dist-test-slave-69rg krb5kdc[26945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 02:26:17.984616 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:18.096804 26958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:18.097097 26958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:18.097187 26958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:18.101184 26958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:18.101276 26958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:18.101315 26958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:18.101375 26958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:18.101424 26958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:18.106720 26958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:18.107818 26958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:18.108855 26958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:18.117304 26966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:18.117475 26963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:18.117511 26964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:18.117584 26958 server_base.cc:1034] running on GCE node
I20260807 02:26:18.118429 26958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:18.119639 26958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:18.120843 26958 hybrid_clock.cc:648] HybridClock initialized: now 1786069578120815 us; error 50 us; skew 500 ppm
Aug 07 02:26:18 dist-test-slave-69rg krb5kdc[26945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:18.123795 26958 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 02:26:18.124723 26958 webserver.cc:460] Webserver started at http://127.0.0.1:42041/ using document root <none> and password file <none>
I20260807 02:26:18.125278 26958 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:18.125370 26958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:18.125612 26958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:18.127764 26958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "5f1f0cc3f3d1425f99670b6e54cfab21"
format_stamp: "Formatted at 2026-08-07 02:26:18 on dist-test-slave-69rg"
server_key: "5f2597786ef94ac43a61efe4c403814d"
server_key_iv: "60602ad3a697398ef9b5596a4728de45"
server_key_version: "encryptionkey@0"
I20260807 02:26:18.128353 26958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "5f1f0cc3f3d1425f99670b6e54cfab21"
format_stamp: "Formatted at 2026-08-07 02:26:18 on dist-test-slave-69rg"
server_key: "5f2597786ef94ac43a61efe4c403814d"
server_key_iv: "60602ad3a697398ef9b5596a4728de45"
server_key_version: "encryptionkey@0"
I20260807 02:26:18.132099 26958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 02:26:18.134523 26974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:18.135754 26958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:26:18.135919 26958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "5f1f0cc3f3d1425f99670b6e54cfab21"
format_stamp: "Formatted at 2026-08-07 02:26:18 on dist-test-slave-69rg"
server_key: "5f2597786ef94ac43a61efe4c403814d"
server_key_iv: "60602ad3a697398ef9b5596a4728de45"
server_key_version: "encryptionkey@0"
I20260807 02:26:18.136049 26958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:18.141144 26958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:18.144771 26958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:18.144991 26958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:18.152658 26958 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39171
I20260807 02:26:18.152669 27026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39171 every 8 connection(s)
I20260807 02:26:18.153631 26958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:18.156381 27027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:18.160780 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 26958
I20260807 02:26:18.160943 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:18.161263 4981 external_mini_cluster.cc:1428] Setting key 750fbd5244d360ee104bc5ceee29ab67
I20260807 02:26:18.162149 27027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21: Bootstrap starting.
I20260807 02:26:18.164505 27027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:18.165432 27027 log.cc:824] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:18.166499 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:18.162619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:35795 (local address 127.0.0.1:39171)
0807 02:26:18.163059 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:18.163074 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:18.163107 (+ 33us) server_negotiation.cc:407] Connection header received
0807 02:26:18.163771 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:18.163785 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:18.164035 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:18.164318 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:18.165156 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:18.165161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:18.165171 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:18.165181 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:18.165230 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:18.165248 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:18.165329 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:18.165476 (+ 147us) server_negotiation.cc:299] Negotiation successful
0807 02:26:18.165647 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":130,"threads_started":1}
I20260807 02:26:18.167397 27027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21: No bootstrap required, opened a new log
I20260807 02:26:18.169355 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:18.171510 27027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1f0cc3f3d1425f99670b6e54cfab21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } }
I20260807 02:26:18.171803 27027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:18.171916 27027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1f0cc3f3d1425f99670b6e54cfab21, State: Initialized, Role: FOLLOWER
I20260807 02:26:18.172396 27027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1f0cc3f3d1425f99670b6e54cfab21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } }
I20260807 02:26:18.172590 27027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:18.172678 27027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:18.172786 27027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:18.173125 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:18.171475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:51253 (local address 127.0.0.1:39171)
0807 02:26:18.171589 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:18.171593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:18.171796 (+ 203us) server_negotiation.cc:407] Connection header received
0807 02:26:18.171866 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:18.171872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:18.171914 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:18.172016 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:18.172681 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:18.172687 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:18.172691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:18.172711 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:18.172760 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:18.172776 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:18.172888 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:18.172972 (+ 84us) server_negotiation.cc:299] Negotiation successful
0807 02:26:18.173003 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 02:26:18.174024 27027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1f0cc3f3d1425f99670b6e54cfab21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } }
I20260807 02:26:18.174360 27027 leader_election.cc:302] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f1f0cc3f3d1425f99670b6e54cfab21; no voters:
I20260807 02:26:18.174652 27027 leader_election.cc:288] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:18.174791 27037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:18.175027 27037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [term 1 LEADER]: Becoming Leader. State: Replica: 5f1f0cc3f3d1425f99670b6e54cfab21, State: Running, Role: LEADER
I20260807 02:26:18.175417 27037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1f0cc3f3d1425f99670b6e54cfab21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } }
I20260807 02:26:18.175822 27027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:18.176949 27039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f1f0cc3f3d1425f99670b6e54cfab21. Latest consensus state: current_term: 1 leader_uuid: "5f1f0cc3f3d1425f99670b6e54cfab21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1f0cc3f3d1425f99670b6e54cfab21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } } }
I20260807 02:26:18.177078 27039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:18.177474 27038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f1f0cc3f3d1425f99670b6e54cfab21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f1f0cc3f3d1425f99670b6e54cfab21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } } }
I20260807 02:26:18.177628 27038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:18.177542 27046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:18.180310 27046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:18.184875 27046 catalog_manager.cc:1347] Generated new cluster ID: ca16d055e3184d91851effcdacebb915
I20260807 02:26:18.184937 27046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:18.194852 27046 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:18.195132 27046 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:18.201630 27046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f1f0cc3f3d1425f99670b6e54cfab21: Generated new TSK 0
I20260807 02:26:18.202266 27046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:18.210562 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 26958
2026-08-07T02:26:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:18 dist-test-slave-69rg krb5kdc[27058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:18 dist-test-slave-69rg krb5kdc[27058](info): set up 2 sockets
Aug 07 02:26:18 dist-test-slave-69rg krb5kdc[27058](info): commencing operation
krb5kdc: starting...
W20260807 02:26:19.269358 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:19 dist-test-slave-69rg krb5kdc[27058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 02:26:19.401749 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:19.515767 27072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:19.516032 27072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:19.516073 27072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:19.520073 27072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:19.520135 27072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:19.520155 27072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:19.520172 27072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:19.520187 27072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:19.525300 27072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:19.526376 27072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:19.527333 27072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:19.535774 27080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:19.535822 27077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:19.535787 27078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:19.536113 27072 server_base.cc:1034] running on GCE node
I20260807 02:26:19.536602 27072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:19.537653 27072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:19.538851 27072 hybrid_clock.cc:648] HybridClock initialized: now 1786069579538824 us; error 38 us; skew 500 ppm
Aug 07 02:26:19 dist-test-slave-69rg krb5kdc[27058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:19.541656 27072 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 02:26:19.542563 27072 webserver.cc:460] Webserver started at http://127.0.0.1:33933/ using document root <none> and password file <none>
I20260807 02:26:19.543113 27072 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:19.543217 27072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:19.543448 27072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:19.545428 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "0fba9c48c0034f88a1498e77befda37c"
format_stamp: "Formatted at 2026-08-07 02:26:19 on dist-test-slave-69rg"
server_key: "0e3fde75d6082c108bad97710679166f"
server_key_iv: "ec45bb83daef7fb6b43c88365b0b3759"
server_key_version: "encryptionkey@0"
I20260807 02:26:19.546057 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "0fba9c48c0034f88a1498e77befda37c"
format_stamp: "Formatted at 2026-08-07 02:26:19 on dist-test-slave-69rg"
server_key: "0e3fde75d6082c108bad97710679166f"
server_key_iv: "ec45bb83daef7fb6b43c88365b0b3759"
server_key_version: "encryptionkey@0"
I20260807 02:26:19.549595 27072 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260807 02:26:19.551956 27087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:19.553224 27072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:26:19.553376 27072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "0fba9c48c0034f88a1498e77befda37c"
format_stamp: "Formatted at 2026-08-07 02:26:19 on dist-test-slave-69rg"
server_key: "0e3fde75d6082c108bad97710679166f"
server_key_iv: "ec45bb83daef7fb6b43c88365b0b3759"
server_key_version: "encryptionkey@0"
I20260807 02:26:19.553493 27072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:19.572458 27072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:19.575946 27072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:19.576170 27072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:19.583796 27072 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34323
I20260807 02:26:19.583820 27141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34323 every 8 connection(s)
I20260807 02:26:19.584758 27072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:19.587611 27142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:19.588666 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 27072
I20260807 02:26:19.588838 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:19.589179 4981 external_mini_cluster.cc:1428] Setting key 2415f45ffc22063aa187bd5b2c533c45
I20260807 02:26:19.593683 27142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c: Bootstrap starting.
I20260807 02:26:19.595803 27142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:19.596546 27142 log.cc:824] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c: Log is configured to *not* fsync() on all Append() calls
Aug 07 02:26:19 dist-test-slave-69rg krb5kdc[27058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260807 02:26:19.598750 27142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c: No bootstrap required, opened a new log
I20260807 02:26:19.602623 27142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fba9c48c0034f88a1498e77befda37c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20260807 02:26:19.602893 27142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:19.602975 27142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fba9c48c0034f88a1498e77befda37c, State: Initialized, Role: FOLLOWER
I20260807 02:26:19.603394 27142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fba9c48c0034f88a1498e77befda37c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20260807 02:26:19.603542 27142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:19.603631 27142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:19.603729 27142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:19.603876 27145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:19.590679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:47607 (local address 127.0.0.1:34323)
0807 02:26:19.591107 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:19.591121 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:19.591156 (+ 35us) server_negotiation.cc:407] Connection header received
0807 02:26:19.591821 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:19.591837 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:19.592155 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:19.592488 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:19.593335 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:19.594396 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:19.595256 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:19.595718 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:19.595888 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:19.598786 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:19.598791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:19.598823 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:19.598860 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:19.600585 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:19.601145 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:19.601149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:19.601153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:19.601297 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:19.601657 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:19.601659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:19.601661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:19.602081 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:19.602291 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:19.602563 (+ 272us) server_negotiation.cc:299] Negotiation successful
0807 02:26:19.602794 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":126,"threads_started":1}
I20260807 02:26:19.604844 27142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fba9c48c0034f88a1498e77befda37c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20260807 02:26:19.605226 27142 leader_election.cc:302] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fba9c48c0034f88a1498e77befda37c; no voters:
I20260807 02:26:19.605562 27142 leader_election.cc:288] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:19.605631 27147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:19.605898 27147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [term 1 LEADER]: Becoming Leader. State: Replica: 0fba9c48c0034f88a1498e77befda37c, State: Running, Role: LEADER
I20260807 02:26:19.606221 27147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fba9c48c0034f88a1498e77befda37c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } }
I20260807 02:26:19.606812 27142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:19.606958 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:19.607435 27147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fba9c48c0034f88a1498e77befda37c. Latest consensus state: current_term: 1 leader_uuid: "0fba9c48c0034f88a1498e77befda37c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fba9c48c0034f88a1498e77befda37c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } } }
I20260807 02:26:19.608246 27147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:19.608135 27148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fba9c48c0034f88a1498e77befda37c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fba9c48c0034f88a1498e77befda37c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34323 } } }
I20260807 02:26:19.608578 27148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c [sys.catalog]: This master's current role is: LEADER
W20260807 02:26:19.612099 27166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:26:19.612195 27166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:26:19.612277 27159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:19.613134 27159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:19.618556 27145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:19.609642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:44491 (local address 127.0.0.1:34323)
0807 02:26:19.609758 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:19.609764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:19.610131 (+ 367us) server_negotiation.cc:407] Connection header received
0807 02:26:19.610291 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:19.610294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:19.610346 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:19.610428 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:19.611084 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:19.611602 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:19.612554 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:19.612796 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:19.612904 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:19.614448 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:19.614451 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:19.614454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:19.614478 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:19.616239 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:19.616848 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:19.616851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:19.616853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:19.616943 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:19.617366 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:19.617370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:19.617372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:19.617602 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:19.617731 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:19.618268 (+ 537us) server_negotiation.cc:299] Negotiation successful
0807 02:26:19.618382 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 02:26:19.619244 27159 catalog_manager.cc:1347] Generated new cluster ID: 7e512efb1d5c404f9ecaf7bed44f1b06
I20260807 02:26:19.619360 27159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:19.662001 27159 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:19.663314 27159 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:19.672345 27159 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fba9c48c0034f88a1498e77befda37c: Generated new TSK 0
I20260807 02:26:19.673197 27159 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:19.680007 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 27072
W20260807 02:26:19.686916 27056 connection.cc:535] client connection to 127.0.0.1:34323 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34323): Connection reset by peer (error 104)
2026-08-07T02:26:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:19 dist-test-slave-69rg krb5kdc[27173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:19 dist-test-slave-69rg krb5kdc[27173](info): set up 2 sockets
Aug 07 02:26:19 dist-test-slave-69rg krb5kdc[27173](info): commencing operation
krb5kdc: starting...
W20260807 02:26:20.745365 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:20 dist-test-slave-69rg krb5kdc[27173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 02:26:20.877252 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:20.989322 27186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:20.989637 27186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:20.989732 27186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:20.993842 27186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:20.993911 27186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:20.993932 27186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:20.993954 27186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:20.993970 27186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:20.999246 27186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:21.000224 27186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:21.001174 27186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:21.010015 27192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:21.010007 27194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:21.010075 27186 server_base.cc:1034] running on GCE node
W20260807 02:26:21.010007 27191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:21.010848 27186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:21.011904 27186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:21.013072 27186 hybrid_clock.cc:648] HybridClock initialized: now 1786069581013058 us; error 32 us; skew 500 ppm
Aug 07 02:26:21 dist-test-slave-69rg krb5kdc[27173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:21.020684 27186 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 02:26:21.021620 27186 webserver.cc:460] Webserver started at http://127.0.0.1:34757/ using document root <none> and password file <none>
I20260807 02:26:21.022249 27186 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:21.022317 27186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:21.022560 27186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:21.024636 27186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "d7771390bd574631983483c859112aef"
format_stamp: "Formatted at 2026-08-07 02:26:21 on dist-test-slave-69rg"
server_key: "594f9c27db116a083978888fac1a77bf"
server_key_iv: "b0f179869e98d382c260c1456d7e231a"
server_key_version: "encryptionkey@0"
I20260807 02:26:21.025210 27186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "d7771390bd574631983483c859112aef"
format_stamp: "Formatted at 2026-08-07 02:26:21 on dist-test-slave-69rg"
server_key: "594f9c27db116a083978888fac1a77bf"
server_key_iv: "b0f179869e98d382c260c1456d7e231a"
server_key_version: "encryptionkey@0"
I20260807 02:26:21.028859 27186 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260807 02:26:21.031185 27201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:21.032464 27186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 02:26:21.032572 27186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "d7771390bd574631983483c859112aef"
format_stamp: "Formatted at 2026-08-07 02:26:21 on dist-test-slave-69rg"
server_key: "594f9c27db116a083978888fac1a77bf"
server_key_iv: "b0f179869e98d382c260c1456d7e231a"
server_key_version: "encryptionkey@0"
I20260807 02:26:21.032728 27186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:21.049292 27186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:21.052289 27186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:21.052524 27186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:21.059978 27186 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38475
I20260807 02:26:21.060006 27255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38475 every 8 connection(s)
I20260807 02:26:21.060952 27186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:21.063779 27256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:21.064530 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 27186
I20260807 02:26:21.064693 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:21.064983 4981 external_mini_cluster.cc:1428] Setting key 7365b60df13b40221352a2a586305d95
I20260807 02:26:21.070768 27256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef: Bootstrap starting.
I20260807 02:26:21.073536 27256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:21.074538 27256 log.cc:824] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:21.075618 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:21.066502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:45297 (local address 127.0.0.1:38475)
0807 02:26:21.066968 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:21.066988 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:21.067034 (+ 46us) server_negotiation.cc:407] Connection header received
0807 02:26:21.067862 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:21.067888 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:21.068332 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:21.068700 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:21.069108 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:21.070592 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:21.071589 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:21.072285 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:21.072455 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:21.073470 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:21.073478 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:21.073497 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:21.074076 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:21.074146 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:21.074189 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:21.074341 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:21.074553 (+ 212us) server_negotiation.cc:299] Negotiation successful
0807 02:26:21.074696 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":124,"threads_started":1}
I20260807 02:26:21.076774 27256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef: No bootstrap required, opened a new log
I20260807 02:26:21.078781 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:21.081442 27256 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7771390bd574631983483c859112aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38475 } }
I20260807 02:26:21.081768 27256 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:21.081868 27256 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7771390bd574631983483c859112aef, State: Initialized, Role: FOLLOWER
I20260807 02:26:21.082360 27256 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7771390bd574631983483c859112aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38475 } }
I20260807 02:26:21.082506 27256 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:21.082659 27256 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:21.082770 27256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:21.083806 27256 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7771390bd574631983483c859112aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38475 } }
I20260807 02:26:21.084206 27256 leader_election.cc:302] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7771390bd574631983483c859112aef; no voters:
I20260807 02:26:21.084574 27256 leader_election.cc:288] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:21.084667 27266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:21.084880 27266 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [term 1 LEADER]: Becoming Leader. State: Replica: d7771390bd574631983483c859112aef, State: Running, Role: LEADER
I20260807 02:26:21.085213 27266 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7771390bd574631983483c859112aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38475 } }
I20260807 02:26:21.086081 27256 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:21.086396 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:21.081264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:46583 (local address 127.0.0.1:38475)
0807 02:26:21.081415 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:21.081420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:21.081652 (+ 232us) server_negotiation.cc:407] Connection header received
0807 02:26:21.081798 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:21.081802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:21.081908 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:21.082013 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:21.082365 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:21.083130 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:21.084017 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:21.084311 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:21.084420 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:21.085696 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:21.085703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:21.085706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:21.085720 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:21.085756 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:21.085772 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:21.086040 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:21.086204 (+ 164us) server_negotiation.cc:299] Negotiation successful
0807 02:26:21.086242 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260807 02:26:21.086556 27268 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7771390bd574631983483c859112aef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7771390bd574631983483c859112aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38475 } } }
I20260807 02:26:21.087073 27268 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:21.088086 27267 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7771390bd574631983483c859112aef. Latest consensus state: current_term: 1 leader_uuid: "d7771390bd574631983483c859112aef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7771390bd574631983483c859112aef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38475 } } }
I20260807 02:26:21.088195 27267 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:21.088876 27272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:21.092648 27272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:21.098572 27272 catalog_manager.cc:1347] Generated new cluster ID: 9a7990c7b6b341b5bd4706c859819e44
I20260807 02:26:21.098661 27272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:21.106938 27272 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:21.107937 27272 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:21.119761 27272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7771390bd574631983483c859112aef: Generated new TSK 0
I20260807 02:26:21.120556 27272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:21.132185 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 27186
W20260807 02:26:21.138527 27171 connection.cc:535] client connection to 127.0.0.1:38475 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38475): Connection reset by peer (error 104)
2026-08-07T02:26:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1451 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:21 dist-test-slave-69rg krb5kdc[27287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:21 dist-test-slave-69rg krb5kdc[27287](info): set up 2 sockets
Aug 07 02:26:21 dist-test-slave-69rg krb5kdc[27287](info): commencing operation
krb5kdc: starting...
W20260807 02:26:22.224212 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:22 dist-test-slave-69rg krb5kdc[27287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260807 02:26:22.389081 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:22.519874 27300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:22.520244 27300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:22.520334 27300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:22.524523 27300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:22.524626 27300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:22.524665 27300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:22.524726 27300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:22.524767 27300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:22.530458 27300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39675
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:22.531828 27300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:22.533133 27300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:22.542721 27305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:22.542919 27300 server_base.cc:1034] running on GCE node
W20260807 02:26:22.542744 27308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:22.542721 27306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:22.543838 27300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:22.545133 27300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:22.546320 27300 hybrid_clock.cc:648] HybridClock initialized: now 1786069582546303 us; error 32 us; skew 500 ppm
Aug 07 02:26:22 dist-test-slave-69rg krb5kdc[27287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:22.549327 27300 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260807 02:26:22.550388 27300 webserver.cc:460] Webserver started at http://127.0.0.1:43855/ using document root <none> and password file <none>
I20260807 02:26:22.550984 27300 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:22.551052 27300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:22.551246 27300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:22.554113 27300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "4906ef147be24500932fadd95df1be49"
format_stamp: "Formatted at 2026-08-07 02:26:22 on dist-test-slave-69rg"
server_key: "cc369450eaaa7b59302361048f9c4861"
server_key_iv: "27d20fbc449d323159b6156ab268f261"
server_key_version: "encryptionkey@0"
I20260807 02:26:22.554785 27300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "4906ef147be24500932fadd95df1be49"
format_stamp: "Formatted at 2026-08-07 02:26:22 on dist-test-slave-69rg"
server_key: "cc369450eaaa7b59302361048f9c4861"
server_key_iv: "27d20fbc449d323159b6156ab268f261"
server_key_version: "encryptionkey@0"
I20260807 02:26:22.558997 27300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:26:22.561581 27315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:22.563004 27300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:26:22.563162 27300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "4906ef147be24500932fadd95df1be49"
format_stamp: "Formatted at 2026-08-07 02:26:22 on dist-test-slave-69rg"
server_key: "cc369450eaaa7b59302361048f9c4861"
server_key_iv: "27d20fbc449d323159b6156ab268f261"
server_key_version: "encryptionkey@0"
I20260807 02:26:22.563273 27300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:22.569095 27300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:22.572443 27300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:22.572646 27300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:22.581133 27300 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39675
I20260807 02:26:22.581151 27370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39675 every 8 connection(s)
I20260807 02:26:22.582314 27300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:22.585568 27371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:22.587612 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 27300
I20260807 02:26:22.587786 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:22.588114 4981 external_mini_cluster.cc:1428] Setting key e61cbe7ac08051731a094b2ea5b6624b
I20260807 02:26:22.593686 27371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49: Bootstrap starting.
I20260807 02:26:22.594774 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:22.589898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:52839 (local address 127.0.0.1:39675)
0807 02:26:22.590407 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:22.590429 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:22.590479 (+ 50us) server_negotiation.cc:407] Connection header received
0807 02:26:22.591447 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:22.591477 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:22.591886 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:22.592306 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:22.593126 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:22.593132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:22.593148 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:22.593160 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:22.593231 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:22.593259 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:22.593409 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:22.593477 (+ 68us) server_negotiation.cc:299] Negotiation successful
0807 02:26:22.593658 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":126,"threads_started":1}
I20260807 02:26:22.596750 27371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:22.597757 27371 log.cc:824] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:22.598078 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:22.600065 27371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49: No bootstrap required, opened a new log
I20260807 02:26:22.602526 27374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:22.600626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:45483 (local address 127.0.0.1:39675)
0807 02:26:22.600769 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:22.600773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:22.600971 (+ 198us) server_negotiation.cc:407] Connection header received
0807 02:26:22.601162 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:22.601164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:22.601196 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:22.601284 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:22.602098 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:22.602102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:22.602104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:22.602117 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:22.602153 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:22.602164 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:22.602253 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:22.602344 (+ 91us) server_negotiation.cc:299] Negotiation successful
0807 02:26:22.602390 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 02:26:22.604638 27371 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4906ef147be24500932fadd95df1be49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39675 } }
I20260807 02:26:22.604887 27371 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:22.604961 27371 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4906ef147be24500932fadd95df1be49, State: Initialized, Role: FOLLOWER
I20260807 02:26:22.605453 27371 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4906ef147be24500932fadd95df1be49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39675 } }
I20260807 02:26:22.605584 27371 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:22.605674 27371 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:22.605793 27371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:22.606911 27371 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4906ef147be24500932fadd95df1be49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39675 } }
I20260807 02:26:22.607316 27371 leader_election.cc:302] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4906ef147be24500932fadd95df1be49; no voters:
I20260807 02:26:22.607661 27371 leader_election.cc:288] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:22.607877 27381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:22.608105 27381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [term 1 LEADER]: Becoming Leader. State: Replica: 4906ef147be24500932fadd95df1be49, State: Running, Role: LEADER
I20260807 02:26:22.608517 27381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4906ef147be24500932fadd95df1be49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39675 } }
I20260807 02:26:22.609210 27371 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:22.609881 27381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4906ef147be24500932fadd95df1be49. Latest consensus state: current_term: 1 leader_uuid: "4906ef147be24500932fadd95df1be49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4906ef147be24500932fadd95df1be49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39675 } } }
I20260807 02:26:22.610005 27381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:22.610579 27382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4906ef147be24500932fadd95df1be49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4906ef147be24500932fadd95df1be49" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39675 } } }
I20260807 02:26:22.610642 27387 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:22.610786 27382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:22.614576 27387 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:22.621178 27387 catalog_manager.cc:1347] Generated new cluster ID: 05c7651cc26e48e0b590e8f45e0d5d38
I20260807 02:26:22.621268 27387 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:22.633996 27387 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:22.634523 27387 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:22.647446 27387 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4906ef147be24500932fadd95df1be49: Generated new TSK 0
I20260807 02:26:22.648392 27387 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:22.656443 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 27300
2026-08-07T02:26:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1525 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:22 dist-test-slave-69rg krb5kdc[27402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:22 dist-test-slave-69rg krb5kdc[27402](info): set up 2 sockets
Aug 07 02:26:22 dist-test-slave-69rg krb5kdc[27402](info): commencing operation
krb5kdc: starting...
W20260807 02:26:23.744649 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:23 dist-test-slave-69rg krb5kdc[27402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:26:23.893108 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44601
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:43075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:44601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:24.030315 27415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:24.030808 27415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:24.030920 27415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:24.036078 27415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:24.036266 27415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:24.036311 27415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:24.036347 27415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:24.036410 27415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:24.043609 27415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:43075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:44601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:24.045264 27415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:24.046715 27415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:24.057566 27423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:24.057590 27421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:24.057580 27420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:24.057969 27415 server_base.cc:1034] running on GCE node
I20260807 02:26:24.058873 27415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:24.060428 27415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:24.063721 27415 hybrid_clock.cc:648] HybridClock initialized: now 1786069584063691 us; error 67 us; skew 500 ppm
Aug 07 02:26:24 dist-test-slave-69rg krb5kdc[27402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069584, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:24.068224 27415 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:26:24.069509 27415 webserver.cc:460] Webserver started at http://127.4.221.126:35777/ using document root <none> and password file <none>
I20260807 02:26:24.070371 27415 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:24.070461 27415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:24.070706 27415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:24.073982 27415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "2ba9f6f0958c408ab89d7bc93328744b"
format_stamp: "Formatted at 2026-08-07 02:26:24 on dist-test-slave-69rg"
server_key: "72d41a6ef499db27e0c7ca72b2ca9fc9"
server_key_iv: "901f1d95087e6b3ba06ce4440e93b194"
server_key_version: "encryptionkey@0"
I20260807 02:26:24.074851 27415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "2ba9f6f0958c408ab89d7bc93328744b"
format_stamp: "Formatted at 2026-08-07 02:26:24 on dist-test-slave-69rg"
server_key: "72d41a6ef499db27e0c7ca72b2ca9fc9"
server_key_iv: "901f1d95087e6b3ba06ce4440e93b194"
server_key_version: "encryptionkey@0"
I20260807 02:26:24.080516 27415 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:26:24.084120 27430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:24.086061 27415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260807 02:26:24.086251 27415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "2ba9f6f0958c408ab89d7bc93328744b"
format_stamp: "Formatted at 2026-08-07 02:26:24 on dist-test-slave-69rg"
server_key: "72d41a6ef499db27e0c7ca72b2ca9fc9"
server_key_iv: "901f1d95087e6b3ba06ce4440e93b194"
server_key_version: "encryptionkey@0"
I20260807 02:26:24.086405 27415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:24.105641 27415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:24.109680 27415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:24.109990 27415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:24.119156 27415 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:44601
I20260807 02:26:24.119174 27484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:44601 every 8 connection(s)
I20260807 02:26:24.120375 27415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:24.122711 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 27415
I20260807 02:26:24.122905 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:24.123310 4981 external_mini_cluster.cc:1428] Setting key 58fe3044deb3f10dcaede05898e0b5e3
I20260807 02:26:24.124616 27485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:24.134017 27485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b: Bootstrap starting.
Aug 07 02:26:24 dist-test-slave-69rg krb5kdc[27402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:26:24.137748 27485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:24.138988 27485 log.cc:824] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:24.142112 27485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b: No bootstrap required, opened a new log
I20260807 02:26:24.142853 27488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:24.125163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58153 (local address 127.4.221.126:44601)
0807 02:26:24.125734 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:24.125759 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:24.125812 (+ 53us) server_negotiation.cc:407] Connection header received
0807 02:26:24.126922 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:24.126953 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:24.127498 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:24.127979 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:24.128959 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:24.130766 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:24.131777 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:24.132674 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:24.132861 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:24.136415 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:24.136424 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:24.136444 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:24.136485 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:24.138890 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:24.139558 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:24.139565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:24.139573 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:24.139717 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:24.140178 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:24.140181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:24.140183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:24.140775 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:24.141019 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:24.141332 (+ 313us) server_negotiation.cc:299] Negotiation successful
0807 02:26:24.141613 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":197,"threads_started":1}
I20260807 02:26:24.146548 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:24.147521 27485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba9f6f0958c408ab89d7bc93328744b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44601 } }
I20260807 02:26:24.147836 27485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:24.147903 27485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba9f6f0958c408ab89d7bc93328744b, State: Initialized, Role: FOLLOWER
I20260807 02:26:24.148499 27485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba9f6f0958c408ab89d7bc93328744b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44601 } }
I20260807 02:26:24.148684 27485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:24.148787 27485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:24.148936 27485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:24.150559 27485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba9f6f0958c408ab89d7bc93328744b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44601 } }
I20260807 02:26:24.151024 27485 leader_election.cc:302] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ba9f6f0958c408ab89d7bc93328744b; no voters:
I20260807 02:26:24.151479 27485 leader_election.cc:288] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:24.151610 27495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:24.151855 27495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [term 1 LEADER]: Becoming Leader. State: Replica: 2ba9f6f0958c408ab89d7bc93328744b, State: Running, Role: LEADER
I20260807 02:26:24.152208 27495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba9f6f0958c408ab89d7bc93328744b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44601 } }
I20260807 02:26:24.153263 27485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:24.153623 27495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ba9f6f0958c408ab89d7bc93328744b. Latest consensus state: current_term: 1 leader_uuid: "2ba9f6f0958c408ab89d7bc93328744b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba9f6f0958c408ab89d7bc93328744b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44601 } } }
I20260807 02:26:24.153765 27495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:24.154161 27496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ba9f6f0958c408ab89d7bc93328744b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba9f6f0958c408ab89d7bc93328744b" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44601 } } }
I20260807 02:26:24.154287 27496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:24.154271 27499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:24.158403 27499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:24.159519 27488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:24.149026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53743 (local address 127.4.221.126:44601)
0807 02:26:24.149174 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:24.149182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:24.149389 (+ 207us) server_negotiation.cc:407] Connection header received
0807 02:26:24.149503 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:24.149506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:24.149560 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:24.149647 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:24.150382 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:24.151308 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:24.152583 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:24.152957 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:24.153076 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:24.155117 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:24.155123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:24.155128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:24.155159 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:24.156935 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:24.157515 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:24.157521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:24.157523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:24.157624 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:24.158038 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:24.158042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:24.158044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:24.158320 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:24.158491 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:24.159262 (+ 771us) server_negotiation.cc:299] Negotiation successful
0807 02:26:24.159365 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 02:26:24.165189 27499 catalog_manager.cc:1347] Generated new cluster ID: d4dd16504ec34cbd9e7aca2208e7dde8
I20260807 02:26:24.165269 27499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:24.172787 27499 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:24.173956 27499 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:24.179037 27499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ba9f6f0958c408ab89d7bc93328744b: Generated new TSK 0
I20260807 02:26:24.180042 27499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:24.193537 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 27415
W20260807 02:26:24.202232 27400 connection.cc:535] client connection to 127.4.221.126:44601 recv error: Network error: failed to read from TLS socket (remote: 127.4.221.126:44601): Connection reset by peer (error 104)
2026-08-07T02:26:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1538 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:24 dist-test-slave-69rg krb5kdc[27516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:24 dist-test-slave-69rg krb5kdc[27516](info): set up 2 sockets
Aug 07 02:26:24 dist-test-slave-69rg krb5kdc[27516](info): commencing operation
krb5kdc: starting...
W20260807 02:26:25.263880 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:25 dist-test-slave-69rg krb5kdc[27516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:26:25.398437 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:41627
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:34193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:41627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:25.517252 27529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:25.517544 27529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:25.517616 27529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:25.521660 27529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:25.521756 27529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:25.521796 27529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:25.521853 27529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:25.521890 27529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:25.527158 27529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:34193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:41627
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:41627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:25.528223 27529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:25.529279 27529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:25.538002 27534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:25.537997 27535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:25.538251 27529 server_base.cc:1034] running on GCE node
W20260807 02:26:25.538034 27537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:25.538892 27529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:25.539978 27529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:25.541188 27529 hybrid_clock.cc:648] HybridClock initialized: now 1786069585541160 us; error 41 us; skew 500 ppm
Aug 07 02:26:25 dist-test-slave-69rg krb5kdc[27516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069585, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:25.544093 27529 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:26:25.545029 27529 webserver.cc:460] Webserver started at http://127.4.221.126:37419/ using document root <none> and password file <none>
I20260807 02:26:25.545630 27529 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:25.545738 27529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:25.546039 27529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:25.548184 27529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "01a13200c3d3407caa297b56beed7237"
format_stamp: "Formatted at 2026-08-07 02:26:25 on dist-test-slave-69rg"
server_key: "1ff0e26e762d766dd7012769148b4048"
server_key_iv: "578b0b317fb95f2cca86ee223e315147"
server_key_version: "encryptionkey@0"
I20260807 02:26:25.548775 27529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "01a13200c3d3407caa297b56beed7237"
format_stamp: "Formatted at 2026-08-07 02:26:25 on dist-test-slave-69rg"
server_key: "1ff0e26e762d766dd7012769148b4048"
server_key_iv: "578b0b317fb95f2cca86ee223e315147"
server_key_version: "encryptionkey@0"
I20260807 02:26:25.552608 27529 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260807 02:26:25.555119 27544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:25.556419 27529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260807 02:26:25.556559 27529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "01a13200c3d3407caa297b56beed7237"
format_stamp: "Formatted at 2026-08-07 02:26:25 on dist-test-slave-69rg"
server_key: "1ff0e26e762d766dd7012769148b4048"
server_key_iv: "578b0b317fb95f2cca86ee223e315147"
server_key_version: "encryptionkey@0"
I20260807 02:26:25.556670 27529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:25.571412 27529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:25.574507 27529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:25.574713 27529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:25.582007 27529 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:41627
I20260807 02:26:25.582036 27598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:41627 every 8 connection(s)
I20260807 02:26:25.582926 27529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:25.585451 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 27529
I20260807 02:26:25.585628 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:25.585959 27599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:25.586053 4981 external_mini_cluster.cc:1428] Setting key 35dac8445c075c47fd2b0d433ea16a62
I20260807 02:26:25.594290 27599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237: Bootstrap starting.
I20260807 02:26:25.597133 27599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:25.597491 27602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:25.587676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33683 (local address 127.4.221.126:41627)
0807 02:26:25.588193 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:25.588215 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:25.588261 (+ 46us) server_negotiation.cc:407] Connection header received
0807 02:26:25.589415 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:25.589442 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:25.590020 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:25.590473 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:25.590934 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:25.592468 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:25.593419 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:25.594160 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:25.594340 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:25.595289 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:25.595295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:25.595310 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:25.595853 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:25.595921 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:25.595951 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:25.596083 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:25.596300 (+ 217us) server_negotiation.cc:299] Negotiation successful
0807 02:26:25.596435 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":157,"threads_started":1}
I20260807 02:26:25.598187 27599 log.cc:824] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:25.600625 27599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237: No bootstrap required, opened a new log
I20260807 02:26:25.600777 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:25.604846 27599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a13200c3d3407caa297b56beed7237" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 41627 } }
I20260807 02:26:25.605118 27599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:25.605203 27599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a13200c3d3407caa297b56beed7237, State: Initialized, Role: FOLLOWER
I20260807 02:26:25.605702 27599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a13200c3d3407caa297b56beed7237" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 41627 } }
I20260807 02:26:25.605877 27599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:25.605950 27599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:25.606055 27599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:25.607007 27599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a13200c3d3407caa297b56beed7237" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 41627 } }
I20260807 02:26:25.607383 27599 leader_election.cc:302] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01a13200c3d3407caa297b56beed7237; no voters:
I20260807 02:26:25.607627 27602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:25.603413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35495 (local address 127.4.221.126:41627)
0807 02:26:25.603536 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:25.603540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:25.603639 (+ 99us) server_negotiation.cc:407] Connection header received
0807 02:26:25.603808 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:25.603813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:25.603858 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:25.603941 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:25.604268 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:25.604780 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:25.605760 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:25.606155 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:25.606260 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:25.607143 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:25.607147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:25.607149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:25.607162 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:25.607189 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:25.607206 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:25.607304 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:25.607428 (+ 124us) server_negotiation.cc:299] Negotiation successful
0807 02:26:25.607462 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 02:26:25.607766 27599 leader_election.cc:288] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:25.608000 27609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:25.608215 27609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [term 1 LEADER]: Becoming Leader. State: Replica: 01a13200c3d3407caa297b56beed7237, State: Running, Role: LEADER
I20260807 02:26:25.608573 27609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a13200c3d3407caa297b56beed7237" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 41627 } }
I20260807 02:26:25.609148 27599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:25.609886 27609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01a13200c3d3407caa297b56beed7237. Latest consensus state: current_term: 1 leader_uuid: "01a13200c3d3407caa297b56beed7237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a13200c3d3407caa297b56beed7237" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 41627 } } }
I20260807 02:26:25.610098 27609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:25.610323 27612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01a13200c3d3407caa297b56beed7237" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01a13200c3d3407caa297b56beed7237" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 41627 } } }
I20260807 02:26:25.610425 27612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:25.610528 27620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:25.613608 27620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:25.619275 27620 catalog_manager.cc:1347] Generated new cluster ID: 33e4494eea0148d9bbdfba2cc385eae7
I20260807 02:26:25.619354 27620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:25.632925 27620 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:25.633939 27620 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:25.639195 27620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01a13200c3d3407caa297b56beed7237: Generated new TSK 0
I20260807 02:26:25.639947 27620 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:25.650326 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 27529
W20260807 02:26:25.657342 27514 connection.cc:535] client connection to 127.4.221.126:41627 recv error: Network error: failed to read from TLS socket (remote: 127.4.221.126:41627): Connection reset by peer (error 104)
2026-08-07T02:26:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1454 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:25 dist-test-slave-69rg krb5kdc[27634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:25 dist-test-slave-69rg krb5kdc[27634](info): set up 2 sockets
Aug 07 02:26:25 dist-test-slave-69rg krb5kdc[27634](info): commencing operation
krb5kdc: starting...
W20260807 02:26:26.789206 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:26 dist-test-slave-69rg krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:26:26.928917 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44569
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:44569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:27.041843 27647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:27.042176 27647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:27.042261 27647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:27.046277 27647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:27.046528 27647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:27.046633 27647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:27.046676 27647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:27.046713 27647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:27.052363 27647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:44569
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:44569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:27.053471 27647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:27.054550 27647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:27.063094 27652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:27.063107 27655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:27.063155 27647 server_base.cc:1034] running on GCE node
W20260807 02:26:27.063128 27653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:27.063927 27647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:27.064976 27647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:27.066166 27647 hybrid_clock.cc:648] HybridClock initialized: now 1786069587066139 us; error 38 us; skew 500 ppm
Aug 07 02:26:27 dist-test-slave-69rg krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069587, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:27.073788 27647 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:26:27.074800 27647 webserver.cc:460] Webserver started at http://127.4.221.126:39987/ using document root <none> and password file <none>
I20260807 02:26:27.075369 27647 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:27.075462 27647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:27.075699 27647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:27.077732 27647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "16681fa20ce245faa810139a7a9ab681"
format_stamp: "Formatted at 2026-08-07 02:26:27 on dist-test-slave-69rg"
server_key: "c5062f4d6df04a06ac99e194dc5eafbd"
server_key_iv: "63ee4745ceb18ba1464fdd7c7169c5bc"
server_key_version: "encryptionkey@0"
I20260807 02:26:27.078382 27647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "16681fa20ce245faa810139a7a9ab681"
format_stamp: "Formatted at 2026-08-07 02:26:27 on dist-test-slave-69rg"
server_key: "c5062f4d6df04a06ac99e194dc5eafbd"
server_key_iv: "63ee4745ceb18ba1464fdd7c7169c5bc"
server_key_version: "encryptionkey@0"
I20260807 02:26:27.082149 27647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:26:27.084501 27664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:27.085748 27647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260807 02:26:27.085927 27647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "16681fa20ce245faa810139a7a9ab681"
format_stamp: "Formatted at 2026-08-07 02:26:27 on dist-test-slave-69rg"
server_key: "c5062f4d6df04a06ac99e194dc5eafbd"
server_key_iv: "63ee4745ceb18ba1464fdd7c7169c5bc"
server_key_version: "encryptionkey@0"
I20260807 02:26:27.086042 27647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:27.090606 27647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:27.093711 27647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:27.093966 27647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:27.101280 27647 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:44569
I20260807 02:26:27.101293 27717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:44569 every 8 connection(s)
I20260807 02:26:27.102236 27647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:27.104913 27718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:27.106172 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 27647
I20260807 02:26:27.106321 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:27.106642 4981 external_mini_cluster.cc:1428] Setting key ef2c056747da602c86b3cbbef6748597
I20260807 02:26:27.111485 27718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681: Bootstrap starting.
I20260807 02:26:27.111815 27721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:27.108059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42561 (local address 127.4.221.126:44569)
0807 02:26:27.108494 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:27.108510 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:27.108544 (+ 34us) server_negotiation.cc:407] Connection header received
0807 02:26:27.109161 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:27.109176 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:27.109454 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:27.109723 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:27.110551 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:27.110555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:27.110566 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:27.110575 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:27.110625 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:27.110642 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:27.110751 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:27.110876 (+ 125us) server_negotiation.cc:299] Negotiation successful
0807 02:26:27.111047 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":154,"threads_started":1}
I20260807 02:26:27.113922 27718 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:27.114614 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:27.114810 27718 log.cc:824] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:27.117041 27718 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681: No bootstrap required, opened a new log
I20260807 02:26:27.118495 27721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:27.116834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47865 (local address 127.4.221.126:44569)
0807 02:26:27.116939 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:27.116942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:27.117124 (+ 182us) server_negotiation.cc:407] Connection header received
0807 02:26:27.117246 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:27.117249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:27.117278 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:27.117343 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:27.118102 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:27.118105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:27.118117 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:27.118128 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:27.118152 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:27.118162 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:27.118251 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:27.118333 (+ 82us) server_negotiation.cc:299] Negotiation successful
0807 02:26:27.118362 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260807 02:26:27.121129 27718 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16681fa20ce245faa810139a7a9ab681" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44569 } }
I20260807 02:26:27.121387 27718 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:27.121446 27718 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16681fa20ce245faa810139a7a9ab681, State: Initialized, Role: FOLLOWER
I20260807 02:26:27.121877 27718 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16681fa20ce245faa810139a7a9ab681" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44569 } }
I20260807 02:26:27.122032 27718 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:27.122098 27718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:27.122191 27718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:27.123083 27718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16681fa20ce245faa810139a7a9ab681" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44569 } }
I20260807 02:26:27.123413 27718 leader_election.cc:302] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16681fa20ce245faa810139a7a9ab681; no voters:
I20260807 02:26:27.123703 27718 leader_election.cc:288] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:27.123858 27728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:27.124171 27728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [term 1 LEADER]: Becoming Leader. State: Replica: 16681fa20ce245faa810139a7a9ab681, State: Running, Role: LEADER
I20260807 02:26:27.124519 27728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16681fa20ce245faa810139a7a9ab681" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44569 } }
I20260807 02:26:27.124939 27718 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:27.126184 27729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16681fa20ce245faa810139a7a9ab681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16681fa20ce245faa810139a7a9ab681" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44569 } } }
I20260807 02:26:27.126307 27729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:27.126658 27731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16681fa20ce245faa810139a7a9ab681. Latest consensus state: current_term: 1 leader_uuid: "16681fa20ce245faa810139a7a9ab681" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16681fa20ce245faa810139a7a9ab681" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 44569 } } }
I20260807 02:26:27.126715 27739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:27.126770 27731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:27.129549 27739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:27.135283 27739 catalog_manager.cc:1347] Generated new cluster ID: b7f74482186648b0aa40fd7333bd8660
I20260807 02:26:27.135351 27739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:27.143059 27739 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:27.143349 27739 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:27.151041 27739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16681fa20ce245faa810139a7a9ab681: Generated new TSK 0
I20260807 02:26:27.151772 27739 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:27.164634 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 27647
2026-08-07T02:26:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1514 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:27 dist-test-slave-69rg krb5kdc[27752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:27 dist-test-slave-69rg krb5kdc[27752](info): set up 2 sockets
Aug 07 02:26:27 dist-test-slave-69rg krb5kdc[27752](info): commencing operation
krb5kdc: starting...
W20260807 02:26:28.241245 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:28 dist-test-slave-69rg krb5kdc[27752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:26:28.369243 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:34427
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:40463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:34427
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:28.480315 27765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:28.480616 27765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:28.480710 27765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:28.484753 27765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:28.484840 27765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:28.484889 27765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:28.484946 27765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:28.484982 27765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:28.490434 27765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:40463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:34427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:34427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:28.491498 27765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:28.492511 27765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:28.501370 27773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:28.501358 27770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:28.501540 27765 server_base.cc:1034] running on GCE node
W20260807 02:26:28.501362 27771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:28.502238 27765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:28.503314 27765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:28.504516 27765 hybrid_clock.cc:648] HybridClock initialized: now 1786069588504489 us; error 39 us; skew 500 ppm
Aug 07 02:26:28 dist-test-slave-69rg krb5kdc[27752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069588, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:28.507401 27765 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:26:28.508273 27765 webserver.cc:460] Webserver started at http://127.4.221.126:37537/ using document root <none> and password file <none>
I20260807 02:26:28.508888 27765 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:28.508991 27765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:28.509233 27765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:28.511417 27765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "ebfb32b516dd4befad2bf085239dd426"
format_stamp: "Formatted at 2026-08-07 02:26:28 on dist-test-slave-69rg"
server_key: "eedfee8a8e5b1c6f600bd527ee1bac8b"
server_key_iv: "c3dc940cdd5df2def74150f5686bb80e"
server_key_version: "encryptionkey@0"
I20260807 02:26:28.512013 27765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "ebfb32b516dd4befad2bf085239dd426"
format_stamp: "Formatted at 2026-08-07 02:26:28 on dist-test-slave-69rg"
server_key: "eedfee8a8e5b1c6f600bd527ee1bac8b"
server_key_iv: "c3dc940cdd5df2def74150f5686bb80e"
server_key_version: "encryptionkey@0"
I20260807 02:26:28.515796 27765 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260807 02:26:28.518164 27780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:28.519367 27765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 02:26:28.519515 27765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "ebfb32b516dd4befad2bf085239dd426"
format_stamp: "Formatted at 2026-08-07 02:26:28 on dist-test-slave-69rg"
server_key: "eedfee8a8e5b1c6f600bd527ee1bac8b"
server_key_iv: "c3dc940cdd5df2def74150f5686bb80e"
server_key_version: "encryptionkey@0"
I20260807 02:26:28.519626 27765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:28.530704 27765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:28.533972 27765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:28.534190 27765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:28.541704 27765 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:34427
I20260807 02:26:28.541720 27834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:34427 every 8 connection(s)
I20260807 02:26:28.542701 27765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:28.545524 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 27765
I20260807 02:26:28.545560 27835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:28.545680 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:28.546015 4981 external_mini_cluster.cc:1428] Setting key c4f5c4a0a47136454a21ff0dc43186a1
I20260807 02:26:28.553435 27835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426: Bootstrap starting.
Aug 07 02:26:28 dist-test-slave-69rg krb5kdc[27752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:26:28.556434 27835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:28.557235 27835 log.cc:824] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:28.559357 27835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426: No bootstrap required, opened a new log
I20260807 02:26:28.561415 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:28.547508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49637 (local address 127.4.221.126:34427)
0807 02:26:28.547941 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:28.547956 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:28.547991 (+ 35us) server_negotiation.cc:407] Connection header received
0807 02:26:28.548597 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:28.548612 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:28.548938 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:28.549205 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:28.550278 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:28.551284 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:28.552650 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:28.553108 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:28.553231 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:28.556485 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:28.556490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:28.556504 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:28.556531 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:28.558504 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:28.558976 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:28.558982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:28.558987 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:28.559100 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:28.559396 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:28.559399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:28.559401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:28.559842 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:28.560061 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:28.560353 (+ 292us) server_negotiation.cc:299] Negotiation successful
0807 02:26:28.560656 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":151,"threads_started":1}
I20260807 02:26:28.563422 27835 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfb32b516dd4befad2bf085239dd426" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34427 } }
I20260807 02:26:28.563751 27835 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:28.563849 27835 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfb32b516dd4befad2bf085239dd426, State: Initialized, Role: FOLLOWER
I20260807 02:26:28.564313 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:28.564316 27835 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfb32b516dd4befad2bf085239dd426" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34427 } }
I20260807 02:26:28.564500 27835 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:28.564599 27835 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:28.564706 27835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:28.565761 27835 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfb32b516dd4befad2bf085239dd426" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34427 } }
I20260807 02:26:28.566179 27835 leader_election.cc:302] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebfb32b516dd4befad2bf085239dd426; no voters:
I20260807 02:26:28.566557 27835 leader_election.cc:288] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:28.566947 27847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:28.567153 27847 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [term 1 LEADER]: Becoming Leader. State: Replica: ebfb32b516dd4befad2bf085239dd426, State: Running, Role: LEADER
I20260807 02:26:28.567451 27847 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfb32b516dd4befad2bf085239dd426" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34427 } }
I20260807 02:26:28.567840 27835 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:28.568511 27847 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebfb32b516dd4befad2bf085239dd426. Latest consensus state: current_term: 1 leader_uuid: "ebfb32b516dd4befad2bf085239dd426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfb32b516dd4befad2bf085239dd426" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34427 } } }
I20260807 02:26:28.568717 27847 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:28.568951 27848 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebfb32b516dd4befad2bf085239dd426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebfb32b516dd4befad2bf085239dd426" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34427 } } }
I20260807 02:26:28.569041 27848 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426 [sys.catalog]: This master's current role is: LEADER
W20260807 02:26:28.573216 27863 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:26:28.573304 27863 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:26:28.573377 27856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:28.574362 27856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:28.579377 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:28.566484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52151 (local address 127.4.221.126:34427)
0807 02:26:28.566604 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:28.566609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:28.570651 (+ 4042us) server_negotiation.cc:407] Connection header received
0807 02:26:28.570813 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:28.570816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:28.570866 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:28.570950 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:28.571908 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:28.572784 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:28.573912 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:28.574160 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:28.574279 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:28.575725 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:28.575729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:28.575732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:28.575764 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:28.577263 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:28.577703 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:28.577707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:28.577709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:28.577792 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:28.578163 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:28.578167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:28.578169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:28.578426 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:28.578587 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:28.579170 (+ 583us) server_negotiation.cc:299] Negotiation successful
0807 02:26:28.579267 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260807 02:26:28.580327 27856 catalog_manager.cc:1347] Generated new cluster ID: d1e02140bc2245a7ab11ca3a167c260b
I20260807 02:26:28.580386 27856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:28.586342 27856 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:28.587324 27856 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:28.596061 27856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebfb32b516dd4befad2bf085239dd426: Generated new TSK 0
I20260807 02:26:28.596791 27856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:28.606056 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 27765
2026-08-07T02:26:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:28 dist-test-slave-69rg krb5kdc[27868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:28 dist-test-slave-69rg krb5kdc[27868](info): set up 2 sockets
Aug 07 02:26:28 dist-test-slave-69rg krb5kdc[27868](info): commencing operation
krb5kdc: starting...
W20260807 02:26:29.663727 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:29 dist-test-slave-69rg krb5kdc[27868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:26:29.793905 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:34553
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:43671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:34553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:29.925159 27881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:29.925557 27881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:29.925647 27881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:29.930398 27881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:29.930511 27881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:29.930548 27881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:29.930590 27881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:29.930627 27881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:29.936762 27881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:43671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:34553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:34553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:29.938233 27881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:29.939534 27881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:29.949424 27887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:29.949431 27889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:29.949573 27881 server_base.cc:1034] running on GCE node
W20260807 02:26:29.949779 27886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:29.950484 27881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:29.951772 27881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:29.952996 27881 hybrid_clock.cc:648] HybridClock initialized: now 1786069589952976 us; error 31 us; skew 500 ppm
Aug 07 02:26:29 dist-test-slave-69rg krb5kdc[27868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069589, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:29.956216 27881 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:26:29.957273 27881 webserver.cc:460] Webserver started at http://127.4.221.126:41315/ using document root <none> and password file <none>
I20260807 02:26:29.957993 27881 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:29.958099 27881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:29.958391 27881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:29.960742 27881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "325fad6ea67d4f2fb9ea6d0699b73836"
format_stamp: "Formatted at 2026-08-07 02:26:29 on dist-test-slave-69rg"
server_key: "cb7ece57d610c8edbde92d16ec64f37f"
server_key_iv: "f7861a0e75cfd3ce8386ea7b85b30cdf"
server_key_version: "encryptionkey@0"
I20260807 02:26:29.961373 27881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "325fad6ea67d4f2fb9ea6d0699b73836"
format_stamp: "Formatted at 2026-08-07 02:26:29 on dist-test-slave-69rg"
server_key: "cb7ece57d610c8edbde92d16ec64f37f"
server_key_iv: "f7861a0e75cfd3ce8386ea7b85b30cdf"
server_key_version: "encryptionkey@0"
I20260807 02:26:29.965636 27881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260807 02:26:29.968379 27896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:29.969978 27881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:26:29.970163 27881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "325fad6ea67d4f2fb9ea6d0699b73836"
format_stamp: "Formatted at 2026-08-07 02:26:29 on dist-test-slave-69rg"
server_key: "cb7ece57d610c8edbde92d16ec64f37f"
server_key_iv: "f7861a0e75cfd3ce8386ea7b85b30cdf"
server_key_version: "encryptionkey@0"
I20260807 02:26:29.970304 27881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:29.991219 27881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:29.995141 27881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:29.995399 27881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:30.004930 27881 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:34553
I20260807 02:26:30.005014 27949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:34553 every 8 connection(s)
I20260807 02:26:30.006307 27881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:30.010936 27950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:30.012368 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 27881
I20260807 02:26:30.012578 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:30.013033 4981 external_mini_cluster.cc:1428] Setting key e154e47dfc3ae2c797c3073cc64ed955
I20260807 02:26:30.020183 27950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836: Bootstrap starting.
I20260807 02:26:30.023746 27950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:30.024842 27950 log.cc:824] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:30.025246 27954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:30.015021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46007 (local address 127.4.221.126:34553)
0807 02:26:30.015593 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:30.015612 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:30.015661 (+ 49us) server_negotiation.cc:407] Connection header received
0807 02:26:30.016773 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:30.016817 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:30.017507 (+ 690us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:30.017998 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:30.018450 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:30.019596 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:30.020915 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:30.021518 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:30.021675 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:30.022911 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:30.022917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:30.022934 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:30.023464 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:30.023540 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:30.023568 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:30.023659 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:30.023904 (+ 245us) server_negotiation.cc:299] Negotiation successful
0807 02:26:30.024180 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":180,"threads_started":1}
I20260807 02:26:30.027500 27950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836: No bootstrap required, opened a new log
I20260807 02:26:30.028745 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:30.032580 27950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325fad6ea67d4f2fb9ea6d0699b73836" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34553 } }
I20260807 02:26:30.032961 27950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:30.033025 27950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 325fad6ea67d4f2fb9ea6d0699b73836, State: Initialized, Role: FOLLOWER
I20260807 02:26:30.033612 27950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325fad6ea67d4f2fb9ea6d0699b73836" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34553 } }
I20260807 02:26:30.033804 27950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:30.033957 27950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:30.034090 27950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:30.035583 27950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325fad6ea67d4f2fb9ea6d0699b73836" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34553 } }
I20260807 02:26:30.036155 27954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:30.031351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56427 (local address 127.4.221.126:34553)
0807 02:26:30.031524 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:30.031527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:30.031659 (+ 132us) server_negotiation.cc:407] Connection header received
0807 02:26:30.031797 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:30.031800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:30.031848 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:30.031920 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:30.032277 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:30.032894 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:30.034029 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:30.034429 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:30.034550 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:30.035515 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:30.035520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:30.035523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:30.035542 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:30.035583 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:30.035601 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:30.035690 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:30.035909 (+ 219us) server_negotiation.cc:299] Negotiation successful
0807 02:26:30.035986 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260807 02:26:30.036123 27950 leader_election.cc:302] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 325fad6ea67d4f2fb9ea6d0699b73836; no voters:
I20260807 02:26:30.036661 27950 leader_election.cc:288] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:30.036851 27961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:30.037117 27961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [term 1 LEADER]: Becoming Leader. State: Replica: 325fad6ea67d4f2fb9ea6d0699b73836, State: Running, Role: LEADER
I20260807 02:26:30.037617 27961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325fad6ea67d4f2fb9ea6d0699b73836" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34553 } }
I20260807 02:26:30.038426 27950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:30.039331 27961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 325fad6ea67d4f2fb9ea6d0699b73836. Latest consensus state: current_term: 1 leader_uuid: "325fad6ea67d4f2fb9ea6d0699b73836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325fad6ea67d4f2fb9ea6d0699b73836" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34553 } } }
I20260807 02:26:30.039484 27961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:30.039860 27962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "325fad6ea67d4f2fb9ea6d0699b73836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "325fad6ea67d4f2fb9ea6d0699b73836" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 34553 } } }
I20260807 02:26:30.039983 27962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836 [sys.catalog]: This master's current role is: LEADER
W20260807 02:26:30.044768 27976 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:26:30.044862 27976 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:26:30.044953 27970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:30.045884 27970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:30.052960 27970 catalog_manager.cc:1347] Generated new cluster ID: f6ed5090e7a247beb4f2a267845305d6
I20260807 02:26:30.053081 27970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:30.069227 27970 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:30.070909 27970 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:30.083300 27970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 325fad6ea67d4f2fb9ea6d0699b73836: Generated new TSK 0
I20260807 02:26:30.084622 27970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:30.093154 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 27881
2026-08-07T02:26:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1488 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:30 dist-test-slave-69rg krb5kdc[27982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:30 dist-test-slave-69rg krb5kdc[27982](info): set up 2 sockets
Aug 07 02:26:30 dist-test-slave-69rg krb5kdc[27982](info): commencing operation
krb5kdc: starting...
W20260807 02:26:31.178663 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:31 dist-test-slave-69rg krb5kdc[27982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:26:31.328797 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:37027
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:37027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:31.457996 27997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:31.458420 27997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:31.458509 27997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:31.462844 27997 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:31.462971 27997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:31.463009 27997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:31.463049 27997 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:31.463083 27997 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:31.468891 27997 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:37027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:37027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:31.470242 27997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:31.471431 27997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:31.481192 28002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:31.481186 28005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:31.481192 28003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:31.481487 27997 server_base.cc:1034] running on GCE node
I20260807 02:26:31.482165 27997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:31.483446 27997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:31.484649 27997 hybrid_clock.cc:648] HybridClock initialized: now 1786069591484638 us; error 40 us; skew 500 ppm
Aug 07 02:26:31 dist-test-slave-69rg krb5kdc[27982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069591, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:31.487928 27997 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:26:31.489086 27997 webserver.cc:460] Webserver started at http://127.4.221.126:46785/ using document root <none> and password file <none>
I20260807 02:26:31.489799 27997 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:31.489975 27997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:31.490267 27997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:31.492708 27997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "9c73e0697fa442428a803b77d63de6e4"
format_stamp: "Formatted at 2026-08-07 02:26:31 on dist-test-slave-69rg"
server_key: "ad73501bb50c47b029c0e88b7d7e3fb9"
server_key_iv: "966957b1d8acccd2b4076eb7196d31a5"
server_key_version: "encryptionkey@0"
I20260807 02:26:31.493386 27997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "9c73e0697fa442428a803b77d63de6e4"
format_stamp: "Formatted at 2026-08-07 02:26:31 on dist-test-slave-69rg"
server_key: "ad73501bb50c47b029c0e88b7d7e3fb9"
server_key_iv: "966957b1d8acccd2b4076eb7196d31a5"
server_key_version: "encryptionkey@0"
I20260807 02:26:31.497993 27997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260807 02:26:31.500934 28012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:31.502533 27997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:26:31.502734 27997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "9c73e0697fa442428a803b77d63de6e4"
format_stamp: "Formatted at 2026-08-07 02:26:31 on dist-test-slave-69rg"
server_key: "ad73501bb50c47b029c0e88b7d7e3fb9"
server_key_iv: "966957b1d8acccd2b4076eb7196d31a5"
server_key_version: "encryptionkey@0"
I20260807 02:26:31.502921 27997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:31.508116 27997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:31.511701 27997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:31.512038 27997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:31.520242 27997 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:37027
I20260807 02:26:31.520270 28065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:37027 every 8 connection(s)
I20260807 02:26:31.521313 27997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:31.524528 28066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:31.527635 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 27997
I20260807 02:26:31.527809 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:31.528126 4981 external_mini_cluster.cc:1428] Setting key 87597a319f266d9a03eac2a157541593
I20260807 02:26:31.531903 28066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4: Bootstrap starting.
I20260807 02:26:31.534126 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:31.529886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52121 (local address 127.4.221.126:37027)
0807 02:26:31.530425 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:31.530442 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:31.530482 (+ 40us) server_negotiation.cc:407] Connection header received
0807 02:26:31.531184 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:31.531205 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:31.531505 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:31.531901 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:31.532778 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:31.532784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:31.532795 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:31.532807 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:31.532863 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:31.532885 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:31.533050 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:31.533122 (+ 72us) server_negotiation.cc:299] Negotiation successful
0807 02:26:31.533291 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":167,"threads_started":1}
I20260807 02:26:31.534902 28066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:31.535780 28066 log.cc:824] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:31.537523 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:31.538318 28066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4: No bootstrap required, opened a new log
I20260807 02:26:31.541885 28069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:31.540083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35307 (local address 127.4.221.126:37027)
0807 02:26:31.540239 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:31.540244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:31.540431 (+ 187us) server_negotiation.cc:407] Connection header received
0807 02:26:31.540559 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:31.540561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:31.540595 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:31.540685 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:31.541444 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:31.541446 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:31.541448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0807 02:26:31.541458 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:31.541483 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0807 02:26:31.541493 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:31.541566 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:31.541686 (+ 120us) server_negotiation.cc:299] Negotiation successful
0807 02:26:31.541720 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260807 02:26:31.543054 28066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c73e0697fa442428a803b77d63de6e4" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37027 } }
I20260807 02:26:31.543341 28066 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260807 02:26:31.543416 28066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c73e0697fa442428a803b77d63de6e4, State: Initialized, Role: FOLLOWER
I20260807 02:26:31.543859 28066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c73e0697fa442428a803b77d63de6e4" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37027 } }
I20260807 02:26:31.544016 28066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260807 02:26:31.544091 28066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260807 02:26:31.544207 28066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:31.545403 28066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c73e0697fa442428a803b77d63de6e4" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37027 } }
I20260807 02:26:31.545781 28066 leader_election.cc:302] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c73e0697fa442428a803b77d63de6e4; no voters:
I20260807 02:26:31.546185 28066 leader_election.cc:288] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260807 02:26:31.546312 28076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:31.546574 28076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [term 1 LEADER]: Becoming Leader. State: Replica: 9c73e0697fa442428a803b77d63de6e4, State: Running, Role: LEADER
I20260807 02:26:31.546939 28076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c73e0697fa442428a803b77d63de6e4" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37027 } }
I20260807 02:26:31.547509 28066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:31.548866 28077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c73e0697fa442428a803b77d63de6e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c73e0697fa442428a803b77d63de6e4" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37027 } } }
I20260807 02:26:31.548856 28078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c73e0697fa442428a803b77d63de6e4. Latest consensus state: current_term: 1 leader_uuid: "9c73e0697fa442428a803b77d63de6e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c73e0697fa442428a803b77d63de6e4" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 37027 } } }
I20260807 02:26:31.549005 28077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:31.549005 28078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:31.550326 28087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:31.553740 28087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:31.559558 28087 catalog_manager.cc:1347] Generated new cluster ID: a1a981dec2cc47cba1b7587b2785ec3d
I20260807 02:26:31.559652 28087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:31.576638 28087 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:31.577059 28087 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:31.585618 28087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c73e0697fa442428a803b77d63de6e4: Generated new TSK 0
I20260807 02:26:31.586575 28087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260807 02:26:31.597009 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 27997
2026-08-07T02:26:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1501 ms)
[----------] 18 tests from AuthTokenIssuingTest (26558 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:31 dist-test-slave-69rg krb5kdc[28098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:31 dist-test-slave-69rg krb5kdc[28098](info): set up 2 sockets
Aug 07 02:26:31 dist-test-slave-69rg krb5kdc[28098](info): commencing operation
krb5kdc: starting...
W20260807 02:26:32.685810 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:32 dist-test-slave-69rg krb5kdc[28098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:26:32.813180 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:43771
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:43771,127.4.221.125:46815,127.4.221.124:45055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:32.938819 28112 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 02:26:32.939185 28112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:32.939213 28112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:32.939316 28112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:32.943596 28112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:32.943678 28112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:32.943698 28112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:32.943717 28112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:32.943732 28112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:32.949546 28112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:43771,127.4.221.125:46815,127.4.221.124:45055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:43771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:32.950978 28112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:32.952266 28112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:32.961948 28117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:32.961987 28120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:32.961987 28118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:32.962343 28112 server_base.cc:1034] running on GCE node
I20260807 02:26:32.962975 28112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:32.964313 28112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:32.965516 28112 hybrid_clock.cc:648] HybridClock initialized: now 1786069592965502 us; error 38 us; skew 500 ppm
Aug 07 02:26:32 dist-test-slave-69rg krb5kdc[28098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069592, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:32.968650 28112 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:26:32.969681 28112 webserver.cc:460] Webserver started at http://127.4.221.126:38831/ using document root <none> and password file <none>
I20260807 02:26:32.970409 28112 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:32.970511 28112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:32.970788 28112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:32.973238 28112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "d3d254cae3f4439181d0441767b7285f"
format_stamp: "Formatted at 2026-08-07 02:26:32 on dist-test-slave-69rg"
server_key: "6bed9636c497fc9b79060548bee376ac"
server_key_iv: "91ac965552f6b25224c9eaad45319092"
server_key_version: "encryptionkey@0"
I20260807 02:26:32.973989 28112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "d3d254cae3f4439181d0441767b7285f"
format_stamp: "Formatted at 2026-08-07 02:26:32 on dist-test-slave-69rg"
server_key: "6bed9636c497fc9b79060548bee376ac"
server_key_iv: "91ac965552f6b25224c9eaad45319092"
server_key_version: "encryptionkey@0"
I20260807 02:26:32.978551 28112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260807 02:26:32.981365 28127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:32.982941 28112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:26:32.983110 28112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "d3d254cae3f4439181d0441767b7285f"
format_stamp: "Formatted at 2026-08-07 02:26:32 on dist-test-slave-69rg"
server_key: "6bed9636c497fc9b79060548bee376ac"
server_key_iv: "91ac965552f6b25224c9eaad45319092"
server_key_version: "encryptionkey@0"
I20260807 02:26:32.983242 28112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:33.002310 28112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:33.006525 28112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:33.006814 28112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:33.016249 28112 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:43771
I20260807 02:26:33.016641 28180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:43771 every 8 connection(s)
I20260807 02:26:33.017509 28112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:33.020643 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 28112
I20260807 02:26:33.020820 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:33.021163 4981 external_mini_cluster.cc:1428] Setting key 41c7bc1ceebdd6b1532c2f6294c95c86
I20260807 02:26:33.021380 28181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:33.026909 28181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.221.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.125@KRBTEST.COM" created.
Aug 07 02:26:33 dist-test-slave-69rg krb5kdc[28098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069592, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:26:33.046165 28184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.029915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45204 (local address 127.4.221.126:43771)
0807 02:26:33.030469 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.030493 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.030779 (+ 286us) server_negotiation.cc:407] Connection header received
0807 02:26:33.032028 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.032033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.032237 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.032446 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.033431 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.034750 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.035864 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.036285 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.036516 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.039991 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:33.039997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:33.040011 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:33.040039 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:33.042211 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.042922 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.042927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.042929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.043058 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.043471 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.043474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.043475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.044001 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:33.044123 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:33.044671 (+ 548us) server_negotiation.cc:299] Negotiation successful
0807 02:26:33.044993 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":364,"spinlock_wait_cycles":1920,"thread_start_us":173,"threads_started":1}
I20260807 02:26:33.046165 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.029915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:43771 (local address 127.0.0.1:45204)
0807 02:26:33.030621 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.030661 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:33.031662 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:33.032442 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:33.032447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:33.032915 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:33.033226 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.033265 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.034888 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.034891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.035732 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.035742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.036465 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.036469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.036656 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.037472 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:33.037492 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:33.039768 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:33.042350 (+ 2582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.042366 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.042382 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.042780 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.043164 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.043168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.043170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.043345 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.044143 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:33.044155 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:33.044363 (+ 208us) client_negotiation.cc:769] Sending connection context
0807 02:26:33.044607 (+ 244us) client_negotiation.cc:240] Negotiation successful
0807 02:26:33.045101 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":147,"spinlock_wait_cycles":44416,"thread_start_us":236,"threads_started":1}
I20260807 02:26:33.049326 28181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:26:33.050331 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.049977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:46815 (local address 127.0.0.1:41468)
0807 02:26:33.050124 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.050219 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 02:26:33.050586 28130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.221.125:46815: connect: Connection refused (error 111)
I20260807 02:26:33.051723 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.051413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:46815 (local address 127.0.0.1:41470)
0807 02:26:33.051571 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.051663 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 02:26:33.051895 28181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.125:46815: Network error: Client connection negotiation failed: client connection to 127.4.221.125:46815: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.125.keytab.
Entry for principal kudu/127.4.221.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.125.keytab.
I20260807 02:26:33.057145 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.125:46815
--webserver_interface=127.4.221.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.125.keytab
--principal=kudu/127.4.221.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:43771,127.4.221.125:46815,127.4.221.124:45055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:26:33.101220 28181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } attempt: 1
I20260807 02:26:33.102551 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.102132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:46815 (local address 127.0.0.1:41480)
0807 02:26:33.102333 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.102420 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 02:26:33.103390 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.103140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:46815 (local address 127.0.0.1:41494)
0807 02:26:33.103253 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.103315 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260807 02:26:33.103564 28181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.125:46815: Network error: Client connection negotiation failed: client connection to 127.4.221.125:46815: connect: Connection refused (error 111)
I20260807 02:26:33.171810 28181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } attempt: 2
I20260807 02:26:33.173043 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.172640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:46815 (local address 127.0.0.1:41510)
0807 02:26:33.172843 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.172922 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 02:26:33.174011 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.173693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:46815 (local address 127.0.0.1:41526)
0807 02:26:33.173862 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.173926 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 02:26:33.174206 28181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.125:46815: Network error: Client connection negotiation failed: client connection to 127.4.221.125:46815: connect: Connection refused (error 111)
W20260807 02:26:33.190657 28188 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 02:26:33.191035 28188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:33.191062 28188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:33.191105 28188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:33.195429 28188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:33.195509 28188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:33.195530 28188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:33.195549 28188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:33.195564 28188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:33.201565 28188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:43771,127.4.221.125:46815,127.4.221.124:45055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.125.keytab
--principal=kudu/127.4.221.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.125:46815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.221.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:33.203086 28188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:33.204335 28188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:33.213881 28196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:33.213902 28194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:33.213881 28193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:33.214275 28188 server_base.cc:1034] running on GCE node
I20260807 02:26:33.214931 28188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:33.216156 28188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:33.217367 28188 hybrid_clock.cc:648] HybridClock initialized: now 1786069593217350 us; error 37 us; skew 500 ppm
Aug 07 02:26:33 dist-test-slave-69rg krb5kdc[28098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069593, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:33.220806 28188 init.cc:377] Logged in from keytab as kudu/127.4.221.125@KRBTEST.COM (short username kudu)
I20260807 02:26:33.221853 28188 webserver.cc:460] Webserver started at http://127.4.221.125:38539/ using document root <none> and password file <none>
I20260807 02:26:33.222509 28188 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:33.222573 28188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:33.222769 28188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:33.225096 28188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/data/instance:
uuid: "9b1b8feb683449aba59507dfabad9c56"
format_stamp: "Formatted at 2026-08-07 02:26:33 on dist-test-slave-69rg"
server_key: "4088c1e1b2f39e7c6c34ad2021d30b8f"
server_key_iv: "dfdbfe3fdf0de6c994e50abdc99b2c7a"
server_key_version: "encryptionkey@0"
I20260807 02:26:33.225721 28188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/wal/instance:
uuid: "9b1b8feb683449aba59507dfabad9c56"
format_stamp: "Formatted at 2026-08-07 02:26:33 on dist-test-slave-69rg"
server_key: "4088c1e1b2f39e7c6c34ad2021d30b8f"
server_key_iv: "dfdbfe3fdf0de6c994e50abdc99b2c7a"
server_key_version: "encryptionkey@0"
I20260807 02:26:33.230207 28188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260807 02:26:33.232982 28203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:33.234678 28188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260807 02:26:33.234855 28188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/wal
uuid: "9b1b8feb683449aba59507dfabad9c56"
format_stamp: "Formatted at 2026-08-07 02:26:33 on dist-test-slave-69rg"
server_key: "4088c1e1b2f39e7c6c34ad2021d30b8f"
server_key_iv: "dfdbfe3fdf0de6c994e50abdc99b2c7a"
server_key_version: "encryptionkey@0"
I20260807 02:26:33.235000 28188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:33.253167 28188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:33.256932 28188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:33.257170 28188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:33.265399 28181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } attempt: 3
I20260807 02:26:33.265640 28188 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.125:46815
I20260807 02:26:33.266045 28258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.125:46815 every 8 connection(s)
I20260807 02:26:33.267001 28188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/data/info.pb
I20260807 02:26:33.270577 28260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 07 02:26:33 dist-test-slave-69rg krb5kdc[28098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069592, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for kudu/127.4.221.125@KRBTEST.COM
I20260807 02:26:33.275588 28260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:26:33.276925 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 28188
I20260807 02:26:33.277081 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-1/wal/instance
I20260807 02:26:33.277408 4981 external_mini_cluster.cc:1428] Setting key 6aa2ebcb98d9b456461e870a0bf921a5
I20260807 02:26:33.280547 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.266180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:46815 (local address 127.0.0.1:41530)
0807 02:26:33.266372 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.266392 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:33.266518 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:33.269343 (+ 2825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:33.269347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:33.269621 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:33.269807 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.269846 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.271363 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.271368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.272008 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.272019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.272817 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.272821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.272933 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.273568 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:33.273583 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:33.275314 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:33.278096 (+ 2782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.278100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.278102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.278433 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.278873 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.278879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.278881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.278984 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.279870 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:33.279876 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:33.280023 (+ 147us) client_negotiation.cc:769] Sending connection context
0807 02:26:33.280157 (+ 134us) client_negotiation.cc:240] Negotiation successful
0807 02:26:33.280338 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260807 02:26:33.282394 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.267122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.4.221.125:46815)
0807 02:26:33.267703 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.267734 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.267780 (+ 46us) server_negotiation.cc:407] Connection header received
0807 02:26:33.268573 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.268592 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.268994 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.269398 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.270011 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.271215 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.272141 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.272670 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.272866 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.275562 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:33.275569 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:33.275588 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:33.275625 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:33.277962 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.278575 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.278581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.278587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.278753 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.279114 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.279120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.279123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.279754 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:33.279896 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:33.280884 (+ 988us) server_negotiation.cc:299] Negotiation successful
0807 02:26:33.281263 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":168,"threads_started":1}
I20260807 02:26:33.284940 28181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:26:33 dist-test-slave-69rg krb5kdc[28098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069593, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.125@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:26:33.285799 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.285520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:42972)
0807 02:26:33.285653 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.285721 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260807 02:26:33.286659 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.286390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:42978)
0807 02:26:33.286532 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.286584 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260807 02:26:33.286864 28181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:45055: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.4.221.124@KRBTEST.COM; defaulting to no policy
I20260807 02:26:33.290737 28184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.278443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45220 (local address 127.4.221.126:43771)
0807 02:26:33.278645 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.278651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.279131 (+ 480us) server_negotiation.cc:407] Connection header received
0807 02:26:33.279381 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.279388 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.279457 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.279557 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.280481 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.281291 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.282413 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.283358 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.283487 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.286593 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:33.286596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:33.286599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:33.286620 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:33.288651 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.289253 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.289255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.289257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.289330 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.289689 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.289692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.289693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.289966 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:33.290050 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:33.290458 (+ 408us) server_negotiation.cc:299] Negotiation successful
0807 02:26:33.290566 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260807 02:26:33.290832 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.278382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:43771 (local address 127.0.0.1:45220)
0807 02:26:33.278990 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.279055 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:33.279223 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:33.279612 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:33.279618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:33.280130 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:33.280345 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.280353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.281430 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.281435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.282277 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.282288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.283499 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.283503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.283647 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.284307 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:33.284321 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:33.286416 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:33.288775 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.288787 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.288796 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.289148 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.289422 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.289426 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.289427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.289543 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.290102 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:33.290107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:33.290231 (+ 124us) client_negotiation.cc:769] Sending connection context
0807 02:26:33.290376 (+ 145us) client_negotiation.cc:240] Negotiation successful
0807 02:26:33.290560 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":181,"threads_started":1}
I20260807 02:26:33.292212 28260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.124@KRBTEST.COM" created.
Aug 07 02:26:33 dist-test-slave-69rg krb5kdc[28098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069593, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.125@KRBTEST.COM for kudu/127.4.221.125@KRBTEST.COM
I20260807 02:26:33.302806 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.292945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41542 (local address 127.4.221.125:46815)
0807 02:26:33.293953 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.293956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.293968 (+ 12us) server_negotiation.cc:407] Connection header received
0807 02:26:33.293998 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.294000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.294035 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.294129 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.294527 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.295219 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.296016 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.296424 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.296518 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.299126 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:33.299130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:33.299133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:33.299162 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:33.301000 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.301461 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.301466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.301468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.301547 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.301857 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.301863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.301868 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.302081 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:33.302193 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:33.302453 (+ 260us) server_negotiation.cc:299] Negotiation successful
0807 02:26:33.302659 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":929}
I20260807 02:26:33.302806 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.292803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:46815 (local address 127.0.0.1:41542)
0807 02:26:33.292949 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.292965 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:33.293051 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:33.294137 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:33.294140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:33.294315 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:33.294418 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.294423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.295343 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.295347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.295899 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.295907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.296654 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.296659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.296791 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.297325 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:33.297344 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:33.298972 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:33.301140 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.301144 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.301146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.301351 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.301638 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.301643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.301645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.301733 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.302180 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:33.302183 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:33.302280 (+ 97us) client_negotiation.cc:769] Sending connection context
0807 02:26:33.302405 (+ 125us) client_negotiation.cc:240] Negotiation successful
0807 02:26:33.302578 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260807 02:26:33.303779 28260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:26:33.304771 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.304429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:42982)
0807 02:26:33.304571 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.304678 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 02:26:33.305573 28206 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
I20260807 02:26:33.306697 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.306487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:42986)
0807 02:26:33.306593 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.306638 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260807 02:26:33.306888 28260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:45055: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.124.keytab.
Entry for principal kudu/127.4.221.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.124.keytab.
I20260807 02:26:33.313432 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.124:45055
--webserver_interface=127.4.221.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.124.keytab
--principal=kudu/127.4.221.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:36689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:43771,127.4.221.125:46815,127.4.221.124:45055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:26:33.319271 28181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } attempt: 1
I20260807 02:26:33.320566 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.320055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:42988)
0807 02:26:33.320270 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.320366 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260807 02:26:33.321413 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.321164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:42998)
0807 02:26:33.321278 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.321342 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 02:26:33.321558 28181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:45055: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
I20260807 02:26:33.356263 28260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } attempt: 1
I20260807 02:26:33.357474 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.357092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:43000)
0807 02:26:33.357284 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.357365 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 02:26:33.358467 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.358152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:43014)
0807 02:26:33.358308 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.358388 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260807 02:26:33.358738 28260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:45055: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
I20260807 02:26:33.396755 28181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } attempt: 2
I20260807 02:26:33.398036 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.397575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:43026)
0807 02:26:33.397764 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.397906 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260807 02:26:33.398900 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.398673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:43042)
0807 02:26:33.398784 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.398840 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260807 02:26:33.399055 28181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:45055: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
I20260807 02:26:33.426951 28260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } attempt: 2
I20260807 02:26:33.428081 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.427705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:43054)
0807 02:26:33.427887 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.427981 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260807 02:26:33.428979 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.428737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:43066)
0807 02:26:33.428852 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.428903 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 02:26:33.429152 28260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:45055: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
W20260807 02:26:33.446862 28266 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 02:26:33.447247 28266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:33.447304 28266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:33.447384 28266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:33.451776 28266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:33.451845 28266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:33.451866 28266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:33.451884 28266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:33.451941 28266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:33.457779 28266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:36689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:43771,127.4.221.125:46815,127.4.221.124:45055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.124.keytab
--principal=kudu/127.4.221.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.124:45055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.221.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:33.459200 28266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:33.460484 28266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:33.470197 28272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:33.470323 28266 server_base.cc:1034] running on GCE node
W20260807 02:26:33.470197 28274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:33.470229 28271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:33.471212 28266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:33.472630 28266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:33.473873 28266 hybrid_clock.cc:648] HybridClock initialized: now 1786069593473843 us; error 48 us; skew 500 ppm
Aug 07 02:26:33 dist-test-slave-69rg krb5kdc[28098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069593, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:33.477098 28266 init.cc:377] Logged in from keytab as kudu/127.4.221.124@KRBTEST.COM (short username kudu)
I20260807 02:26:33.478415 28266 webserver.cc:460] Webserver started at http://127.4.221.124:41643/ using document root <none> and password file <none>
I20260807 02:26:33.479108 28266 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:33.479207 28266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:33.479509 28266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:33.482235 28266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/data/instance:
uuid: "c2ab07b3db974f02b19a2618328074ec"
format_stamp: "Formatted at 2026-08-07 02:26:33 on dist-test-slave-69rg"
server_key: "a42e344ca2405e2f900c5b28a0ce6f93"
server_key_iv: "8d39e8e8ed86353479bfc5e689728bf5"
server_key_version: "encryptionkey@0"
I20260807 02:26:33.482915 28266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/wal/instance:
uuid: "c2ab07b3db974f02b19a2618328074ec"
format_stamp: "Formatted at 2026-08-07 02:26:33 on dist-test-slave-69rg"
server_key: "a42e344ca2405e2f900c5b28a0ce6f93"
server_key_iv: "8d39e8e8ed86353479bfc5e689728bf5"
server_key_version: "encryptionkey@0"
I20260807 02:26:33.487773 28266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260807 02:26:33.490876 28281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:33.492556 28266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:26:33.492725 28266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/wal
uuid: "c2ab07b3db974f02b19a2618328074ec"
format_stamp: "Formatted at 2026-08-07 02:26:33 on dist-test-slave-69rg"
server_key: "a42e344ca2405e2f900c5b28a0ce6f93"
server_key_iv: "8d39e8e8ed86353479bfc5e689728bf5"
server_key_version: "encryptionkey@0"
I20260807 02:26:33.492869 28266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:33.498235 28181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } attempt: 3
I20260807 02:26:33.499313 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.498961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:43074)
0807 02:26:33.499131 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.499209 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260807 02:26:33.500186 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.499944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:43080)
0807 02:26:33.500070 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.500121 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260807 02:26:33.500392 28181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:45055: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
I20260807 02:26:33.514798 28266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:33.518594 28266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:33.518834 28266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:33.520349 28260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } attempt: 3
I20260807 02:26:33.521334 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.521007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:43082)
0807 02:26:33.521168 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.521239 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260807 02:26:33.522215 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.521953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:43096)
0807 02:26:33.522106 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.522159 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260807 02:26:33.522338 28260 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:45055: Network error: Client connection negotiation failed: client connection to 127.4.221.124:45055: connect: Connection refused (error 111)
I20260807 02:26:33.527662 28266 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.124:45055
I20260807 02:26:33.527985 28334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.124:45055 every 8 connection(s)
I20260807 02:26:33.528880 28266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/data/info.pb
I20260807 02:26:33.532366 28335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:33.532738 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 28266
I20260807 02:26:33.532889 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786069301823597-4981-0/minicluster-data/master-2/wal/instance
I20260807 02:26:33.533232 4981 external_mini_cluster.cc:1428] Setting key 8e041e66886a7405ba2671028ae445b9
I20260807 02:26:33.538101 28335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:26:33 dist-test-slave-69rg krb5kdc[28098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.124@KRBTEST.COM
Aug 07 02:26:33 dist-test-slave-69rg krb5kdc[28098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069593, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.124@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:26:33.551031 28338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.534849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43100 (local address 127.4.221.124:45055)
0807 02:26:33.535354 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.535372 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.535451 (+ 79us) server_negotiation.cc:407] Connection header received
0807 02:26:33.536270 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.536293 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.536696 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.537096 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.538130 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.539375 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.540521 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.541283 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.541420 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.544931 (+ 3511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:33.544937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:33.544949 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:33.544981 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:33.547004 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.547565 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.547571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.547579 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.547737 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.548125 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.548128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.548129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.548614 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:33.548778 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:33.549241 (+ 463us) server_negotiation.cc:299] Negotiation successful
0807 02:26:33.549648 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":139,"threads_started":1}
I20260807 02:26:33.551716 28184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.540497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45232 (local address 127.4.221.126:43771)
0807 02:26:33.540698 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.540702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.541099 (+ 397us) server_negotiation.cc:407] Connection header received
0807 02:26:33.541277 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.541281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.541369 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.541472 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.542497 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.543361 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.544694 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.545038 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.545126 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.548027 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:33.548031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:33.548034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:33.548071 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:33.549801 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.550334 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.550336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.550338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.550458 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.550736 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.550739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.550740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.550942 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:33.551021 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:33.551439 (+ 418us) server_negotiation.cc:299] Negotiation successful
0807 02:26:33.551582 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260807 02:26:33.551743 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.540372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:43771 (local address 127.0.0.1:45232)
0807 02:26:33.540950 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.540996 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:33.541163 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:33.541547 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:33.541553 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:33.542143 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:33.542346 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.542353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.543535 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.543538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.544562 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.544573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.545156 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.545159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.545302 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.545834 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:33.545859 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:33.547862 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:33.549946 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.549954 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.549960 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.550238 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.550549 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.550552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.550554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.550647 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.551052 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:33.551061 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:33.551199 (+ 138us) client_negotiation.cc:769] Sending connection context
0807 02:26:33.551356 (+ 157us) client_negotiation.cc:240] Negotiation successful
0807 02:26:33.551557 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":162,"threads_started":1}
I20260807 02:26:33.553440 28335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:26:33 dist-test-slave-69rg krb5kdc[28098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069593, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.124@KRBTEST.COM for kudu/127.4.221.125@KRBTEST.COM
Aug 07 02:26:33 dist-test-slave-69rg krb5kdc[28098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:26:33.564906 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.554086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:46815 (local address 127.0.0.1:41556)
0807 02:26:33.554517 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.554532 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:33.554616 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:33.554892 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:33.554896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:33.555096 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:33.555273 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.555282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.556391 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.556394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.557273 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.557284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.557998 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.558001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.558140 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.558827 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:33.558853 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:33.560614 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:33.562957 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.562962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.562965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.563184 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.563532 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.563535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.563537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.563654 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.564324 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:33.564331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:33.564463 (+ 132us) client_negotiation.cc:769] Sending connection context
0807 02:26:33.564578 (+ 115us) client_negotiation.cc:240] Negotiation successful
0807 02:26:33.564742 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353}
I20260807 02:26:33.565155 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.554258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41556 (local address 127.4.221.125:46815)
0807 02:26:33.554413 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.554419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.554608 (+ 189us) server_negotiation.cc:407] Connection header received
0807 02:26:33.554705 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.554710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.554776 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.554855 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.555443 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.556272 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.557452 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.557863 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.557976 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.560796 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:33.560800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:33.560802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:33.560835 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:33.562826 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.563295 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.563301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.563303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.563405 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.563824 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.563828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.563830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.564181 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:33.564328 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:33.564685 (+ 357us) server_negotiation.cc:299] Negotiation successful
0807 02:26:33.564873 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260807 02:26:33.566165 28184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.554237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.4.221.126:43771)
0807 02:26:33.554356 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.554360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.554448 (+ 88us) server_negotiation.cc:407] Connection header received
0807 02:26:33.554601 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.554605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.554660 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.554748 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.555626 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.556365 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.557494 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.557861 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.557974 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.561492 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:33.561499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:33.561506 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:33.561549 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:33.563546 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.564108 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.564113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.564115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.564235 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.564613 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.564617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.564620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.564916 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:33.565266 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:33.565685 (+ 419us) server_negotiation.cc:299] Negotiation successful
0807 02:26:33.565906 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260807 02:26:33.566247 28335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:26:33 dist-test-slave-69rg krb5kdc[28098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069593, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.124@KRBTEST.COM for kudu/127.4.221.124@KRBTEST.COM
Aug 07 02:26:33 dist-test-slave-69rg krb5kdc[28098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.125@KRBTEST.COM
I20260807 02:26:33.577206 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.567002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:43108)
0807 02:26:33.567163 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.567407 (+ 244us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:33.567489 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:33.567873 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:33.567877 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:33.568091 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:33.568228 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.568236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.569283 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.569287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.570167 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.570177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.570801 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.570805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.570925 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.571550 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:33.571570 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:33.573338 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:33.575553 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.575557 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.575559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.575771 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.576058 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.576062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.576064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.576147 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.576588 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:33.576591 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:33.576689 (+ 98us) client_negotiation.cc:769] Sending connection context
0807 02:26:33.576795 (+ 106us) client_negotiation.cc:240] Negotiation successful
0807 02:26:33.576985 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 02:26:33.577206 28338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.567088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43108 (local address 127.4.221.124:45055)
0807 02:26:33.567235 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.567241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.567478 (+ 237us) server_negotiation.cc:407] Connection header received
0807 02:26:33.567562 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.567568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.567626 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.567724 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.568364 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.569164 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.570285 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.570670 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.570778 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.573536 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:33.573540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:33.573543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:33.573570 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:33.575413 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.575873 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.575878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.575880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.575967 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.576235 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.576240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.576242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.576481 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:33.576591 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:33.576861 (+ 270us) server_negotiation.cc:299] Negotiation successful
0807 02:26:33.577050 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":63}
I20260807 02:26:33.578876 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.567786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41560 (local address 127.4.221.125:46815)
0807 02:26:33.567943 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.567949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.567969 (+ 20us) server_negotiation.cc:407] Connection header received
0807 02:26:33.568017 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.568024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.568081 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.568185 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.569009 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.569864 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.570952 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.571303 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.571532 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.574670 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:33.574677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:33.574683 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:33.574724 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:33.576550 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.577079 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.577085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.577087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.577197 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.577920 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.577924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.577926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.578161 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:33.578471 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:33.578556 (+ 85us) server_negotiation.cc:299] Negotiation successful
0807 02:26:33.578699 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260807 02:26:33.582340 28335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec: Bootstrap starting.
I20260807 02:26:33.582898 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:33.585348 28335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:33.584632 28154 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d3d254cae3f4439181d0441767b7285f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45242
I20260807 02:26:33.586349 28335 log.cc:824] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:33.588989 28335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec: No bootstrap required, opened a new log
I20260807 02:26:33.593736 28335 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } }
I20260807 02:26:33.594086 28335 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ab07b3db974f02b19a2618328074ec, State: Initialized, Role: FOLLOWER
I20260807 02:26:33.594470 28335 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } }
I20260807 02:26:33.595099 28341 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } } }
I20260807 02:26:33.595265 28341 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:26:33.595976 28335 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec [sys.catalog]: configured and running, proceeding with master startup.
W20260807 02:26:33.602108 28352 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:26:33.602210 28352 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:26:33.600713 28266 master_runner.cc:186] Error getting master registration for 127.4.221.126:43771: OK, instance_id { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" instance_seqno: 1786069593013618 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 02:26:33.604803 28266 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 02:26:33.664613 28181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } attempt: 4
I20260807 02:26:33.665527 28260 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } attempt: 4
Aug 07 02:26:33 dist-test-slave-69rg krb5kdc[28098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069592, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for kudu/127.4.221.124@KRBTEST.COM
Aug 07 02:26:33 dist-test-slave-69rg krb5kdc[28098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069593, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.125@KRBTEST.COM for kudu/127.4.221.124@KRBTEST.COM
I20260807 02:26:33.677407 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.666151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:43132)
0807 02:26:33.666374 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.666389 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:33.666496 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:33.667614 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:33.667617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:33.667807 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:33.667924 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.667930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.669063 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.669068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.670067 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.670082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.670738 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.670742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.670849 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.671337 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:33.671352 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:33.673133 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:33.675624 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.675628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.675631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.675857 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.676200 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.676203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.676205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.676299 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.676754 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:33.676757 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:33.676872 (+ 115us) client_negotiation.cc:769] Sending connection context
0807 02:26:33.676993 (+ 121us) client_negotiation.cc:240] Negotiation successful
0807 02:26:33.677164 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260807 02:26:33.677434 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.666804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43132 (local address 127.4.221.124:45055)
0807 02:26:33.667329 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.667343 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.667358 (+ 15us) server_negotiation.cc:407] Connection header received
0807 02:26:33.667403 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.667405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.667458 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.667567 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.668051 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.668922 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.670211 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.670597 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.670705 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.673283 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:33.673287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:33.673290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:33.673322 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:33.675521 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.675972 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.675979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.675981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.676075 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.676398 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.676404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.676406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.676656 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:33.676758 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:33.677049 (+ 291us) server_negotiation.cc:299] Negotiation successful
0807 02:26:33.677174 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":199,"threads_started":1}
I20260807 02:26:33.677884 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.665547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:45055 (local address 127.0.0.1:43122)
0807 02:26:33.665959 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:33.665983 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:33.666129 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:33.666797 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:33.666801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:33.667113 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:33.667262 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.667270 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.668402 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.668406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.669184 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:33.669193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.669672 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.669675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:33.669764 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.670513 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:33.670533 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:33.673133 (+ 2600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:33.675870 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.675875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.675877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.676179 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.676562 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:33.676566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:33.676568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:33.676671 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:33.677210 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:33.677214 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:33.677335 (+ 121us) client_negotiation.cc:769] Sending connection context
0807 02:26:33.677471 (+ 136us) client_negotiation.cc:240] Negotiation successful
0807 02:26:33.677638 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233}
I20260807 02:26:33.678577 28338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.665704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.4.221.124:45055)
0807 02:26:33.666333 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.666339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.666354 (+ 15us) server_negotiation.cc:407] Connection header received
0807 02:26:33.666419 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.666423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.666477 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.666552 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.667439 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.668280 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.669300 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.669550 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.669666 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.673327 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:33.673332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:33.673335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:33.673380 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:33.675715 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.676314 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.676318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.676320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.676436 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.676791 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.676795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.676797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.677049 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:33.677158 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:33.678250 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0807 02:26:33.678389 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"spinlock_wait_cycles":5376}
I20260807 02:26:33.682065 28181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f: Bootstrap starting.
I20260807 02:26:33.682195 28260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56: Bootstrap starting.
I20260807 02:26:33.685047 28181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:33.685108 28260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:33.686126 28181 log.cc:824] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:33.686125 28260 log.cc:824] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:33.688850 28181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f: No bootstrap required, opened a new log
I20260807 02:26:33.691335 28260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56: No bootstrap required, opened a new log
I20260807 02:26:33.692833 28181 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } }
I20260807 02:26:33.693132 28181 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d254cae3f4439181d0441767b7285f, State: Initialized, Role: FOLLOWER
I20260807 02:26:33.693554 28181 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } }
I20260807 02:26:33.694293 28357 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } } }
I20260807 02:26:33.694593 28357 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:26:33.695261 28181 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:33.695365 28154 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 02:26:33.695497 28154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:33.695637 28260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } }
I20260807 02:26:33.695863 28260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b1b8feb683449aba59507dfabad9c56, State: Initialized, Role: FOLLOWER
I20260807 02:26:33.696187 28260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } }
I20260807 02:26:33.697007 28154 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } }
I20260807 02:26:33.697672 28260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:33.698230 28154 leader_election.cc:288] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [CANDIDATE]: Term 1 election: Requested vote from peers 9b1b8feb683449aba59507dfabad9c56 (127.4.221.125:46815), c2ab07b3db974f02b19a2618328074ec (127.4.221.124:45055)
I20260807 02:26:33.701896 28231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3d254cae3f4439181d0441767b7285f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9b1b8feb683449aba59507dfabad9c56"
I20260807 02:26:33.702149 28231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56 [term 0 FOLLOWER]: Advancing to term 1
W20260807 02:26:33.703115 28372 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:26:33.703197 28372 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:26:33.703821 28231 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3d254cae3f4439181d0441767b7285f in term 1.
I20260807 02:26:33.704027 28358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } } }
I20260807 02:26:33.704181 28358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:26:33.704726 28308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3d254cae3f4439181d0441767b7285f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c2ab07b3db974f02b19a2618328074ec"
I20260807 02:26:33.705041 28308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:33.706221 28130 leader_election.cc:302] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b1b8feb683449aba59507dfabad9c56, d3d254cae3f4439181d0441767b7285f; no voters:
I20260807 02:26:33.706699 28308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3d254cae3f4439181d0441767b7285f in term 1.
I20260807 02:26:33.701891 28112 master_runner.cc:186] Error getting master registration for 127.4.221.125:46815: OK, instance_id { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" instance_seqno: 1786069593263222 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 02:26:33.707306 28112 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 02:26:33.707650 28357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:33.707902 28357 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [term 1 LEADER]: Becoming Leader. State: Replica: d3d254cae3f4439181d0441767b7285f, State: Running, Role: LEADER
I20260807 02:26:33.708338 28357 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } }
W20260807 02:26:33.713653 28389 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:26:33.713745 28389 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:26:33.714841 28357 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3d254cae3f4439181d0441767b7285f. Latest consensus state: current_term: 1 leader_uuid: "d3d254cae3f4439181d0441767b7285f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } } }
I20260807 02:26:33.714982 28357 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:33.715423 28392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:33.716591 28392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:33.723182 28308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec [term 1 FOLLOWER]: Refusing update from remote peer d3d254cae3f4439181d0441767b7285f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:26:33.724614 28357 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:26:33.725137 28231 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56 [term 1 FOLLOWER]: Refusing update from remote peer d3d254cae3f4439181d0441767b7285f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:26:33.727056 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.707419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41568 (local address 127.4.221.125:46815)
0807 02:26:33.707552 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.707557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.715463 (+ 7906us) server_negotiation.cc:407] Connection header received
0807 02:26:33.715711 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.715714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.715792 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.715858 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.716944 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.717713 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.718972 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.719295 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.719395 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.721626 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:33.721630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:33.721633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:33.721661 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:33.723515 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.724375 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.724381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.724384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.724501 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.725935 (+ 1434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.725939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.725940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.726198 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:33.726329 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:33.726734 (+ 405us) server_negotiation.cc:299] Negotiation successful
0807 02:26:33.726874 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 02:26:33.727097 28184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.706737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45258 (local address 127.4.221.126:43771)
0807 02:26:33.706879 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.706885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.709724 (+ 2839us) server_negotiation.cc:407] Connection header received
0807 02:26:33.710035 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.710040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.710108 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.710211 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.711583 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.712497 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.713984 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.714334 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.714432 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.721446 (+ 7014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:33.721450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:33.721453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:33.721481 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:33.725164 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.725701 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.725708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.725711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.725798 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.726137 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.726140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.726142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.726375 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:33.726503 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:33.726825 (+ 322us) server_negotiation.cc:299] Negotiation successful
0807 02:26:33.726961 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260807 02:26:33.727450 28382 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:26:33.730353 28341 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3d254cae3f4439181d0441767b7285f. Latest consensus state: current_term: 1 leader_uuid: "d3d254cae3f4439181d0441767b7285f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } } }
I20260807 02:26:33.730530 28341 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:26:33.733172 28358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3d254cae3f4439181d0441767b7285f. Latest consensus state: current_term: 1 leader_uuid: "d3d254cae3f4439181d0441767b7285f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } } }
I20260807 02:26:33.733325 28358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:26:33.735136 28357 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3d254cae3f4439181d0441767b7285f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } } }
I20260807 02:26:33.735287 28357 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:33.735633 28338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:33.707938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.4.221.124:45055)
0807 02:26:33.708074 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.708079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.719790 (+ 11711us) server_negotiation.cc:407] Connection header received
0807 02:26:33.720025 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.720029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.720090 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.720154 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.721172 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.721979 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.723226 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:33.723578 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:33.723674 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:33.726982 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:33.726987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:33.726991 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:33.727025 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:33.728950 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.734000 (+ 5050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.734006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.734008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.734123 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:33.734489 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:33.734492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:33.734494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:33.734750 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:33.734885 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:33.735286 (+ 401us) server_negotiation.cc:299] Negotiation successful
0807 02:26:33.735464 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260807 02:26:33.736724 28341 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3d254cae3f4439181d0441767b7285f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } } }
I20260807 02:26:33.736860 28341 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:26:33.737536 28393 mvcc.cc:205] Tried to move back new op lower bound from 7315741055881990144 to 7315741055833939968. Current Snapshot: MvccSnapshot[applied={T|T < 7315741055881990144}]
I20260807 02:26:33.739540 28396 mvcc.cc:205] Tried to move back new op lower bound from 7315741055881990144 to 7315741055833939968. Current Snapshot: MvccSnapshot[applied={T|T < 7315741055881990144}]
I20260807 02:26:33.741978 28392 catalog_manager.cc:1347] Generated new cluster ID: 557683379c6741c6884d918a354cc79b
I20260807 02:26:33.742054 28392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:33.754859 28394 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3d254cae3f4439181d0441767b7285f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } } }
I20260807 02:26:33.755091 28394 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:33.758307 28358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3d254cae3f4439181d0441767b7285f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d254cae3f4439181d0441767b7285f" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 43771 } } peers { permanent_uuid: "9b1b8feb683449aba59507dfabad9c56" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 46815 } } peers { permanent_uuid: "c2ab07b3db974f02b19a2618328074ec" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 45055 } } }
I20260807 02:26:33.758452 28358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:26:33.767094 28392 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:33.768731 28392 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:33.791989 28392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3d254cae3f4439181d0441767b7285f: Generated new TSK 0
I20260807 02:26:33.793309 28392 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 02:26:33.838514 28184 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:33.834386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45270 (local address 127.4.221.126:43771)
0807 02:26:33.834573 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.834580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.836778 (+ 2198us) server_negotiation.cc:407] Connection header received
0807 02:26:33.837315 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.837319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.837411 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.837474 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.838291 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45270: BlockingRecv error: recv got EOF from 127.0.0.1:45270 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260807 02:26:33.840584 28259 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:33.835393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.4.221.125:46815)
0807 02:26:33.835528 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.835533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.837205 (+ 1672us) server_negotiation.cc:407] Connection header received
0807 02:26:33.839203 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.839209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.839279 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.839350 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.840415 (+ 1065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41578: BlockingRecv error: recv got EOF from 127.0.0.1:41578 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260807 02:26:33.842445 28338 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:33.836078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.4.221.124:45055)
0807 02:26:33.836212 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.836216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.840052 (+ 3836us) server_negotiation.cc:407] Connection header received
0807 02:26:33.841544 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.841548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.841618 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.841675 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.842284 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43142: BlockingRecv error: recv got EOF from 127.0.0.1:43142 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260807 02:26:33.847978 28259 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:33.846086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.4.221.125:46815)
0807 02:26:33.846217 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.846222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.846857 (+ 635us) server_negotiation.cc:407] Connection header received
0807 02:26:33.847116 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.847119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.847221 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.847276 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.847846 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41582: BlockingRecv error: recv got EOF from 127.0.0.1:41582 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260807 02:26:33.853670 28259 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:33.852025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.4.221.125:46815)
0807 02:26:33.852137 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.852141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.852690 (+ 549us) server_negotiation.cc:407] Connection header received
0807 02:26:33.852879 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.852882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.852940 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.852992 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.853547 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41590: BlockingRecv error: recv got EOF from 127.0.0.1:41590 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260807 02:26:33.859920 28259 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:33.857986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41598 (local address 127.4.221.125:46815)
0807 02:26:33.858099 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.858103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.858719 (+ 616us) server_negotiation.cc:407] Connection header received
0807 02:26:33.858990 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.858996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.859063 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.859120 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.859785 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41598: BlockingRecv error: recv got EOF from 127.0.0.1:41598 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260807 02:26:33.871251 28259 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:33.866750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.4.221.125:46815)
0807 02:26:33.866899 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.866905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.870202 (+ 3297us) server_negotiation.cc:407] Connection header received
0807 02:26:33.870405 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.870410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.870481 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.870580 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.871102 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41604: BlockingRecv error: recv got EOF from 127.0.0.1:41604 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260807 02:26:33.883199 28259 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:33.881755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.4.221.125:46815)
0807 02:26:33.881913 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.881917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.882198 (+ 281us) server_negotiation.cc:407] Connection header received
0807 02:26:33.882352 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.882355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.882422 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.882523 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.883051 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41610: BlockingRecv error: recv got EOF from 127.0.0.1:41610 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260807 02:26:33.903314 28259 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:33.901874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41622 (local address 127.4.221.125:46815)
0807 02:26:33.902052 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.902058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.902206 (+ 148us) server_negotiation.cc:407] Connection header received
0807 02:26:33.902352 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.902357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.902444 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.902559 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.903137 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41622: BlockingRecv error: recv got EOF from 127.0.0.1:41622 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260807 02:26:33.939208 28259 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:33.937724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.4.221.125:46815)
0807 02:26:33.937888 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:33.937893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:33.938082 (+ 189us) server_negotiation.cc:407] Connection header received
0807 02:26:33.938261 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:33.938266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:33.938338 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:33.938445 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:33.939063 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41624: BlockingRecv error: recv got EOF from 127.0.0.1:41624 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260807 02:26:34.006836 28259 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:34.005500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.4.221.125:46815)
0807 02:26:34.005704 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:34.005709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:34.005869 (+ 160us) server_negotiation.cc:407] Connection header received
0807 02:26:34.006082 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:34.006085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:34.006138 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:34.006246 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:34.006715 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41628: BlockingRecv error: recv got EOF from 127.0.0.1:41628 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260807 02:26:34.138435 28259 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:34.137148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41638 (local address 127.4.221.125:46815)
0807 02:26:34.137306 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:34.137309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:34.137455 (+ 146us) server_negotiation.cc:407] Connection header received
0807 02:26:34.137578 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:34.137580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:34.137641 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:34.137726 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:34.138276 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41638: BlockingRecv error: recv got EOF from 127.0.0.1:41638 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260807 02:26:34.398213 28259 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:34.396778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41646 (local address 127.4.221.125:46815)
0807 02:26:34.397004 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:34.397009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:34.397027 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:26:34.397277 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:34.397280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:34.397341 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:34.397441 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:34.398021 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41646: BlockingRecv error: recv got EOF from 127.0.0.1:41646 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260807 02:26:34.603745 28352 catalog_manager.cc:1259] Loaded cluster ID: 557683379c6741c6884d918a354cc79b
I20260807 02:26:34.603839 28352 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec: loading cluster ID for follower catalog manager: success
I20260807 02:26:34.605540 28352 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec: acquiring CA information for follower catalog manager: success
I20260807 02:26:34.606766 28352 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c2ab07b3db974f02b19a2618328074ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 02:26:34.715128 28389 catalog_manager.cc:1259] Loaded cluster ID: 557683379c6741c6884d918a354cc79b
I20260807 02:26:34.715256 28389 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56: loading cluster ID for follower catalog manager: success
I20260807 02:26:34.716981 28389 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56: acquiring CA information for follower catalog manager: success
I20260807 02:26:34.718253 28389 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b1b8feb683449aba59507dfabad9c56: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 02:26:34.916152 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:34.911995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.4.221.125:46815)
0807 02:26:34.912251 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:34.912257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:34.912366 (+ 109us) server_negotiation.cc:407] Connection header received
0807 02:26:34.912537 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:34.912540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:34.912686 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:34.912809 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:26:34.913074 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:34.913663 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:34.914588 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:34.914821 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:34.914937 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:34.915268 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:26:34.915733 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:26:34.915876 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:34.915917 (+ 41us) server_negotiation.cc:299] Negotiation successful
0807 02:26:34.915975 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
I20260807 02:26:34.924979 28477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:34.920326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.4.221.124:45055)
0807 02:26:34.920644 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:34.920648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:34.920664 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:26:34.920775 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:34.920777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:34.920929 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:34.921046 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:26:34.921286 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:34.921873 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:34.922891 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:34.923127 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:34.923205 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:34.923553 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:26:34.924262 (+ 709us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:26:34.924371 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:34.924600 (+ 229us) server_negotiation.cc:299] Negotiation successful
0807 02:26:34.924713 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":140,"threads_started":1}
I20260807 02:26:34.933259 28471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:34.929075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.4.221.125:46815)
0807 02:26:34.929253 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:34.929258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:34.929389 (+ 131us) server_negotiation.cc:407] Connection header received
0807 02:26:34.929593 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:34.929597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:34.929669 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:34.929725 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:26:34.930119 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:34.930805 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:34.931822 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:34.932157 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:34.932266 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:34.932699 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:26:34.932798 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:26:34.932885 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:34.933011 (+ 126us) server_negotiation.cc:299] Negotiation successful
0807 02:26:34.933084 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260807 02:26:34.933468 28477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:34.929767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.4.221.124:45055)
0807 02:26:34.929929 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:34.929932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:34.929943 (+ 11us) server_negotiation.cc:407] Connection header received
0807 02:26:34.929981 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:34.929983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:34.930024 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:34.930108 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:26:34.930456 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:34.931195 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:34.932191 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:34.932494 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:34.932594 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:34.932977 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:26:34.933059 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:26:34.933107 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:34.933275 (+ 168us) server_negotiation.cc:299] Negotiation successful
0807 02:26:34.933341 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260807 02:26:34.933799 28483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:34.928626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.4.221.126:43771)
0807 02:26:34.928928 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:34.928934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:34.928952 (+ 18us) server_negotiation.cc:407] Connection header received
0807 02:26:34.929123 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:34.929126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:34.929363 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:34.929784 (+ 421us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:26:34.929970 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:34.930708 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:34.931857 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:34.932245 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:34.932343 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:34.932703 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:26:34.933120 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:26:34.933509 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:34.933550 (+ 41us) server_negotiation.cc:299] Negotiation successful
0807 02:26:34.933627 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":95,"threads_started":1}
I20260807 02:26:34.937988 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 28112
I20260807 02:26:34.947985 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 28188
I20260807 02:26:34.955632 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 28266
2026-08-07T02:26:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3361 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 07 02:26:35 dist-test-slave-69rg krb5kdc[28488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 07 02:26:35 dist-test-slave-69rg krb5kdc[28488](info): set up 2 sockets
Aug 07 02:26:35 dist-test-slave-69rg krb5kdc[28488](info): commencing operation
krb5kdc: starting...
W20260807 02:26:36.133873 4981 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.000s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/test-user.keytab.
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-07T02:26:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-07T02:26:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.221.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
Entry for principal kudu/127.4.221.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab.
I20260807 02:26:36.262333 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:39367
--webserver_interface=127.4.221.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:39367,127.4.221.125:45121,127.4.221.124:43005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260807 02:26:36.380157 28501 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 02:26:36.380491 28501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:36.380568 28501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:36.380656 28501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:36.384608 28501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:36.384692 28501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:36.384732 28501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:36.384766 28501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:36.384811 28501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:36.389997 28501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:39367,127.4.221.125:45121,127.4.221.124:43005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.126.keytab
--principal=kudu/127.4.221.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.126:39367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.221.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:36.391048 28501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:36.392081 28501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260807 02:26:36.400439 28501 server_base.cc:1034] running on GCE node
W20260807 02:26:36.400420 28506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:36.400419 28509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:36.400631 28507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:36.401252 28501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:36.402354 28501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:36.403524 28501 hybrid_clock.cc:648] HybridClock initialized: now 1786069596403498 us; error 38 us; skew 500 ppm
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:36.406381 28501 init.cc:377] Logged in from keytab as kudu/127.4.221.126@KRBTEST.COM (short username kudu)
I20260807 02:26:36.407261 28501 webserver.cc:460] Webserver started at http://127.4.221.126:40347/ using document root <none> and password file <none>
I20260807 02:26:36.407840 28501 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:36.407932 28501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:36.408188 28501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:36.410717 28501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/data/instance:
uuid: "34d2236078c249948987f746fa8aba35"
format_stamp: "Formatted at 2026-08-07 02:26:36 on dist-test-slave-69rg"
server_key: "66a7f951042fa39b4e75d10690a11d1e"
server_key_iv: "7a2d5d896aa4c0523682a06de69a9ecd"
server_key_version: "encryptionkey@0"
I20260807 02:26:36.411250 28501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/wal/instance:
uuid: "34d2236078c249948987f746fa8aba35"
format_stamp: "Formatted at 2026-08-07 02:26:36 on dist-test-slave-69rg"
server_key: "66a7f951042fa39b4e75d10690a11d1e"
server_key_iv: "7a2d5d896aa4c0523682a06de69a9ecd"
server_key_version: "encryptionkey@0"
I20260807 02:26:36.415016 28501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260807 02:26:36.417454 28516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:36.418926 28501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260807 02:26:36.419211 28501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/wal
uuid: "34d2236078c249948987f746fa8aba35"
format_stamp: "Formatted at 2026-08-07 02:26:36 on dist-test-slave-69rg"
server_key: "66a7f951042fa39b4e75d10690a11d1e"
server_key_iv: "7a2d5d896aa4c0523682a06de69a9ecd"
server_key_version: "encryptionkey@0"
I20260807 02:26:36.419348 28501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:36.435405 28501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:36.438719 28501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:36.438925 28501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:36.446197 28501 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.126:39367
I20260807 02:26:36.446475 28569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.126:39367 every 8 connection(s)
I20260807 02:26:36.447162 28501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/data/info.pb
I20260807 02:26:36.449020 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 28501
I20260807 02:26:36.449147 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-0/wal/instance
I20260807 02:26:36.449470 4981 external_mini_cluster.cc:1428] Setting key 4c8dd37b2e0589b1645ffb2cba8b3734
I20260807 02:26:36.450524 28570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:36.454644 28570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.221.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.125@KRBTEST.COM" created.
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:26:36.470196 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.456923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43212 (local address 127.4.221.126:39367)
0807 02:26:36.457484 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:36.457498 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:36.457548 (+ 50us) server_negotiation.cc:407] Connection header received
0807 02:26:36.458672 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:36.458676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:36.458861 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:36.459014 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:36.459849 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.460828 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.462239 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.462594 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.462791 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.465308 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:36.465314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:36.465323 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:36.465344 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:36.466993 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.467458 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.467468 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.467470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.467566 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.467882 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.467887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.467889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.468285 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:36.468423 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:36.468786 (+ 363us) server_negotiation.cc:299] Negotiation successful
0807 02:26:36.469081 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":442,"thread_start_us":150,"threads_started":1}
I20260807 02:26:36.470196 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.456922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:39367 (local address 127.0.0.1:43212)
0807 02:26:36.457414 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.457465 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:36.458380 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:36.459047 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:36.459051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:36.459474 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:36.459713 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.459728 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.460984 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.460989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.462116 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.462131 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.462765 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.462769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.462875 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.463481 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:36.463496 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:36.465155 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:36.467100 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.467111 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.467120 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.467347 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.467672 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.467675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.467677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.467791 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.468405 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:36.468411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:36.468550 (+ 139us) client_negotiation.cc:769] Sending connection context
0807 02:26:36.468714 (+ 164us) client_negotiation.cc:240] Negotiation successful
0807 02:26:36.469130 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":161,"thread_start_us":139,"threads_started":1}
I20260807 02:26:36.472708 28570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:26:36.473552 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.473197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45121 (local address 127.0.0.1:46902)
0807 02:26:36.473390 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.473487 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:45121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260807 02:26:36.473758 28520 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.221.125:45121: connect: Connection refused (error 111)
I20260807 02:26:36.474754 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.474560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45121 (local address 127.0.0.1:46918)
0807 02:26:36.474655 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.474700 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:45121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260807 02:26:36.474922 28570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.125:45121: Network error: Client connection negotiation failed: client connection to 127.4.221.125:45121: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.125.keytab.
Entry for principal kudu/127.4.221.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.125.keytab.
I20260807 02:26:36.477599 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.125:45121
--webserver_interface=127.4.221.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.125.keytab
--principal=kudu/127.4.221.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:39367,127.4.221.125:45121,127.4.221.124:43005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:26:36.524170 28570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } attempt: 1
I20260807 02:26:36.525063 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.524761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45121 (local address 127.0.0.1:46928)
0807 02:26:36.524924 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.524986 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:45121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260807 02:26:36.525884 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.525588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45121 (local address 127.0.0.1:46942)
0807 02:26:36.525718 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.525766 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:45121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260807 02:26:36.526095 28570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.125:45121: Network error: Client connection negotiation failed: client connection to 127.4.221.125:45121: connect: Connection refused (error 111)
W20260807 02:26:36.589128 28577 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 02:26:36.589454 28577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:36.589529 28577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:36.589609 28577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260807 02:26:36.593683 28577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:36.593757 28577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:36.593796 28577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:36.593870 28577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:36.593906 28577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:36.594278 28570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } attempt: 2
I20260807 02:26:36.595211 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.594871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45121 (local address 127.0.0.1:46954)
0807 02:26:36.595058 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.595126 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:45121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260807 02:26:36.595971 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.595752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45121 (local address 127.0.0.1:46962)
0807 02:26:36.595863 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.595921 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.125:45121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260807 02:26:36.596107 28570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.125:45121: Network error: Client connection negotiation failed: client connection to 127.4.221.125:45121: connect: Connection refused (error 111)
I20260807 02:26:36.599515 28577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:39367,127.4.221.125:45121,127.4.221.124:43005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.125.keytab
--principal=kudu/127.4.221.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.125:45121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.221.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:36.600528 28577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:36.601518 28577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:36.609967 28585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:36.609975 28582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:36.609985 28583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:36.610035 28577 server_base.cc:1034] running on GCE node
I20260807 02:26:36.610778 28577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:36.611832 28577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:36.612993 28577 hybrid_clock.cc:648] HybridClock initialized: now 1786069596612976 us; error 31 us; skew 500 ppm
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:36.615701 28577 init.cc:377] Logged in from keytab as kudu/127.4.221.125@KRBTEST.COM (short username kudu)
I20260807 02:26:36.616525 28577 webserver.cc:460] Webserver started at http://127.4.221.125:46285/ using document root <none> and password file <none>
I20260807 02:26:36.617064 28577 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:36.617125 28577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:36.617367 28577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:36.619417 28577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/data/instance:
uuid: "3800131fd11345dc9e4cced913cd807d"
format_stamp: "Formatted at 2026-08-07 02:26:36 on dist-test-slave-69rg"
server_key: "1238bf1ef0d9bd6bfe6a76bd6daacc89"
server_key_iv: "c05b8c3eaba1e6d665d102c76e2e9e24"
server_key_version: "encryptionkey@0"
I20260807 02:26:36.619979 28577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/wal/instance:
uuid: "3800131fd11345dc9e4cced913cd807d"
format_stamp: "Formatted at 2026-08-07 02:26:36 on dist-test-slave-69rg"
server_key: "1238bf1ef0d9bd6bfe6a76bd6daacc89"
server_key_iv: "c05b8c3eaba1e6d665d102c76e2e9e24"
server_key_version: "encryptionkey@0"
I20260807 02:26:36.623735 28577 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260807 02:26:36.626168 28592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:36.627352 28577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260807 02:26:36.627480 28577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/wal
uuid: "3800131fd11345dc9e4cced913cd807d"
format_stamp: "Formatted at 2026-08-07 02:26:36 on dist-test-slave-69rg"
server_key: "1238bf1ef0d9bd6bfe6a76bd6daacc89"
server_key_iv: "c05b8c3eaba1e6d665d102c76e2e9e24"
server_key_version: "encryptionkey@0"
I20260807 02:26:36.627581 28577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:36.654966 28577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:36.658067 28577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:36.658301 28577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:36.665477 28577 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.125:45121
I20260807 02:26:36.665782 28644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.125:45121 every 8 connection(s)
I20260807 02:26:36.666462 28577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/data/info.pb
I20260807 02:26:36.669122 28645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:36.672785 28645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:26:36.674733 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 28577
I20260807 02:26:36.674862 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-1/wal/instance
I20260807 02:26:36.675171 4981 external_mini_cluster.cc:1428] Setting key 38129534daf39741d4405c974780e6a3
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.125@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.221.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.221.124@KRBTEST.COM" created.
I20260807 02:26:36.686674 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.675117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43214 (local address 127.4.221.126:39367)
0807 02:26:36.675276 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:36.675281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:36.675895 (+ 614us) server_negotiation.cc:407] Connection header received
0807 02:26:36.676839 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:36.676842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:36.676906 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:36.677014 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:36.678257 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.678804 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.679995 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.680234 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.680343 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.682814 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:36.682818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:36.682821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:36.682856 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:36.684347 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.684925 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.684929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.684931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.685018 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.685370 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.685375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.685376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.685592 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:36.685741 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:36.686366 (+ 625us) server_negotiation.cc:299] Negotiation successful
0807 02:26:36.686500 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 02:26:36.687268 28570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } attempt: 3
I20260807 02:26:36.687551 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.675177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:39367 (local address 127.0.0.1:43214)
0807 02:26:36.675723 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.675769 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:36.676630 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:36.677177 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:36.677184 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:36.677545 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:36.678077 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.678091 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.678948 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.678952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.679858 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.679868 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.680394 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.680398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.680566 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.681175 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:36.681192 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:36.682641 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:36.684536 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.684542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.684557 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.684818 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.685112 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.685115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.685117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.685269 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.685730 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:36.685737 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:36.686093 (+ 356us) client_negotiation.cc:769] Sending connection context
0807 02:26:36.686294 (+ 201us) client_negotiation.cc:240] Negotiation successful
0807 02:26:36.686612 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":143,"threads_started":1}
I20260807 02:26:36.689915 28645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for kudu/127.4.221.125@KRBTEST.COM
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.125@KRBTEST.COM for kudu/127.4.221.125@KRBTEST.COM
I20260807 02:26:36.698971 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.687725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45121 (local address 127.0.0.1:46978)
0807 02:26:36.687952 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.687968 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:36.688040 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:36.688722 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:36.688728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:36.688929 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:36.689062 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.689071 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.690440 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.690447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.691980 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.691995 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.692639 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.692643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.692777 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.693263 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:36.693275 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:36.694776 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:36.697058 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.697062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.697065 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.697265 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.697652 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.697658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.697662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.697777 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.698447 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:36.698449 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:36.698536 (+ 87us) client_negotiation.cc:769] Sending connection context
0807 02:26:36.698622 (+ 86us) client_negotiation.cc:240] Negotiation successful
0807 02:26:36.698728 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260807 02:26:36.699342 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.688041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46978 (local address 127.4.221.125:45121)
0807 02:26:36.688331 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:36.688340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:36.688357 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:26:36.688389 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:36.688393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:36.688569 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:36.688706 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:36.689188 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.690305 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.692114 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.692522 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.692638 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.694912 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:36.694922 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:36.694935 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:36.694968 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:36.696944 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.697382 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.697388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.697390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.697499 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.697926 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.697932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.697937 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.698332 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:36.698514 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:36.698735 (+ 221us) server_negotiation.cc:299] Negotiation successful
0807 02:26:36.698930 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":123,"threads_started":1}
I20260807 02:26:36.700439 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.690472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45121 (local address 127.0.0.1:46984)
0807 02:26:36.690588 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.690604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:36.690673 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:36.691194 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:36.691197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:36.691376 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:36.691459 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.691463 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.692224 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.692229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.693224 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.693238 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.693897 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.693901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.694022 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.694519 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:36.694535 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:36.696119 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:36.698807 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.698810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.698812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.699085 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.699404 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.699410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.699411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.699521 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.699925 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:36.699928 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:36.700003 (+ 75us) client_negotiation.cc:769] Sending connection context
0807 02:26:36.700087 (+ 84us) client_negotiation.cc:240] Negotiation successful
0807 02:26:36.700220 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 02:26:36.700471 28651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.690708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.4.221.125:45121)
0807 02:26:36.691011 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:36.691022 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:36.691032 (+ 10us) server_negotiation.cc:407] Connection header received
0807 02:26:36.691058 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:36.691059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:36.691089 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:36.691186 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:36.691566 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.692115 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.693368 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.693732 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.693871 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.696278 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:36.696281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:36.696283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:36.696305 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:36.698691 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.699203 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.699207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.699209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.699294 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.699615 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.699618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.699620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.699816 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:36.699910 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:36.700176 (+ 266us) server_negotiation.cc:299] Negotiation successful
0807 02:26:36.700292 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":136,"threads_started":1}
I20260807 02:26:36.701366 28570 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:26:36.701470 28645 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } has no permanent_uuid. Determining permanent_uuid...
I20260807 02:26:36.702275 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.701995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:43005 (local address 127.0.0.1:43440)
0807 02:26:36.702140 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.702199 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260807 02:26:36.702332 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.702036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:43005 (local address 127.0.0.1:43452)
0807 02:26:36.702189 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.702269 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260807 02:26:36.702515 28595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
I20260807 02:26:36.703420 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.703152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:43005 (local address 127.0.0.1:43466)
0807 02:26:36.703309 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.703360 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260807 02:26:36.703538 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.703299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:43005 (local address 127.0.0.1:43468)
0807 02:26:36.703407 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.703451 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260807 02:26:36.703588 28570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:43005: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
W20260807 02:26:36.703727 28645 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:43005: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.221.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.124.keytab.
Entry for principal kudu/127.4.221.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.124.keytab.
I20260807 02:26:36.705188 4981 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
/tmp/dist-test-taskBkVddJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.124:43005
--webserver_interface=127.4.221.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.124.keytab
--principal=kudu/127.4.221.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.221.84:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.221.126:39367,127.4.221.125:45121,127.4.221.124:43005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260807 02:26:36.734766 28570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } attempt: 1
I20260807 02:26:36.735720 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.735404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:43005 (local address 127.0.0.1:43470)
0807 02:26:36.735569 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.735639 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260807 02:26:36.736441 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.736243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:43005 (local address 127.0.0.1:43478)
0807 02:26:36.736347 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.736392 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 02:26:36.736603 28570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:43005: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
I20260807 02:26:36.753024 28645 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } attempt: 1
I20260807 02:26:36.754025 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.753636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:43005 (local address 127.0.0.1:43482)
0807 02:26:36.753853 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.753929 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260807 02:26:36.754813 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.754575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:43005 (local address 127.0.0.1:43484)
0807 02:26:36.754686 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.754749 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260807 02:26:36.754949 28645 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:43005: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
I20260807 02:26:36.811807 28570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } attempt: 2
I20260807 02:26:36.812776 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.812446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:43005 (local address 127.0.0.1:43488)
0807 02:26:36.812610 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.812690 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260807 02:26:36.813575 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.813352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:43005 (local address 127.0.0.1:43504)
0807 02:26:36.813466 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.813516 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260807 02:26:36.813855 28570 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:43005: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
W20260807 02:26:36.819232 28653 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260807 02:26:36.819511 28653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260807 02:26:36.819585 28653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260807 02:26:36.819657 28653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260807 02:26:36.823093 28645 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } attempt: 2
I20260807 02:26:36.823918 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.823600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:43005 (local address 127.0.0.1:43508)
0807 02:26:36.823769 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.823834 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260807 02:26:36.824227 28653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260807 02:26:36.824290 28653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260807 02:26:36.824317 28653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260807 02:26:36.824344 28653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260807 02:26:36.824360 28653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260807 02:26:36.824836 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.824602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:43005 (local address 127.0.0.1:43514)
0807 02:26:36.824724 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.824776 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260807 02:26:36.824972 28645 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.221.124:43005: Network error: Client connection negotiation failed: client connection to 127.4.221.124:43005: connect: Connection refused (error 111)
I20260807 02:26:36.830087 28653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.221.84:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.221.126:39367,127.4.221.125:45121,127.4.221.124:43005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/krb5kdc/kudu_127.4.221.124.keytab
--principal=kudu/127.4.221.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.221.124:43005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.221.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 07 Aug 2026 01:47:39 UTC on ccycloud.quasar-ogxevj.root.comops.site
build id 4571737
I20260807 02:26:36.831233 28653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260807 02:26:36.832327 28653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260807 02:26:36.840682 28659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:36.840747 28658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260807 02:26:36.840705 28661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260807 02:26:36.841105 28653 server_base.cc:1034] running on GCE node
I20260807 02:26:36.841706 28653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260807 02:26:36.842829 28653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260807 02:26:36.843995 28653 hybrid_clock.cc:648] HybridClock initialized: now 1786069596843983 us; error 38 us; skew 500 ppm
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260807 02:26:36.846905 28653 init.cc:377] Logged in from keytab as kudu/127.4.221.124@KRBTEST.COM (short username kudu)
I20260807 02:26:36.847749 28653 webserver.cc:460] Webserver started at http://127.4.221.124:42955/ using document root <none> and password file <none>
I20260807 02:26:36.848258 28653 fs_manager.cc:359] Metadata directory not provided
I20260807 02:26:36.848316 28653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260807 02:26:36.848479 28653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260807 02:26:36.850531 28653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/data/instance:
uuid: "4acf9c9c9bda435daeee0e7792d3e18a"
format_stamp: "Formatted at 2026-08-07 02:26:36 on dist-test-slave-69rg"
server_key: "4c51b6022cdbc864086ebe36f18ee844"
server_key_iv: "f028623aa8c8f8fccbd8fa74341e419f"
server_key_version: "encryptionkey@0"
I20260807 02:26:36.851023 28653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/wal/instance:
uuid: "4acf9c9c9bda435daeee0e7792d3e18a"
format_stamp: "Formatted at 2026-08-07 02:26:36 on dist-test-slave-69rg"
server_key: "4c51b6022cdbc864086ebe36f18ee844"
server_key_iv: "f028623aa8c8f8fccbd8fa74341e419f"
server_key_version: "encryptionkey@0"
I20260807 02:26:36.854738 28653 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260807 02:26:36.857035 28668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260807 02:26:36.858359 28653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260807 02:26:36.858500 28653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/data,/tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/wal
uuid: "4acf9c9c9bda435daeee0e7792d3e18a"
format_stamp: "Formatted at 2026-08-07 02:26:36 on dist-test-slave-69rg"
server_key: "4c51b6022cdbc864086ebe36f18ee844"
server_key_iv: "f028623aa8c8f8fccbd8fa74341e419f"
server_key_version: "encryptionkey@0"
I20260807 02:26:36.858613 28653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260807 02:26:36.868561 28653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260807 02:26:36.871850 28653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260807 02:26:36.872068 28653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260807 02:26:36.879292 28653 rpc_server.cc:307] RPC server started. Bound to: 127.4.221.124:43005
I20260807 02:26:36.879612 28721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.221.124:43005 every 8 connection(s)
I20260807 02:26:36.880322 28653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/data/info.pb
I20260807 02:26:36.882452 4981 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu as pid 28653
I20260807 02:26:36.882592 4981 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskBkVddJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786069301823597-4981-0/minicluster-data/master-2/wal/instance
I20260807 02:26:36.882925 4981 external_mini_cluster.cc:1428] Setting key 667b9c2806f1e24e2244941cdba4c26e
I20260807 02:26:36.883441 28722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260807 02:26:36.888821 28722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.124@KRBTEST.COM
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.124@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
I20260807 02:26:36.899080 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.884367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43520 (local address 127.4.221.124:43005)
0807 02:26:36.884783 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:36.884804 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:36.884851 (+ 47us) server_negotiation.cc:407] Connection header received
0807 02:26:36.885806 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:36.885855 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:36.886315 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:36.886697 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:36.887549 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.888841 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.889768 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.890340 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.890508 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.893948 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:36.893953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:36.893961 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:36.893987 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:36.895884 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.896344 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.896351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.896357 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.896496 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.896817 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.896820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.896822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.897270 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:36.897466 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:36.897774 (+ 308us) server_negotiation.cc:299] Negotiation successful
0807 02:26:36.898133 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":165,"threads_started":1}
I20260807 02:26:36.900758 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.890586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.126:39367 (local address 127.0.0.1:43228)
0807 02:26:36.891097 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.891145 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:36.891284 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:36.891630 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:36.891635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:36.892171 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:36.892353 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.892364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.893491 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.893498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.894319 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.894329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.894789 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.894793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.894918 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.895406 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:36.895418 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:36.896951 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:36.898857 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.898865 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.898870 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.899154 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.899612 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.899615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.899616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.899710 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.900106 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:36.900115 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:36.900235 (+ 120us) client_negotiation.cc:769] Sending connection context
0807 02:26:36.900378 (+ 143us) client_negotiation.cc:240] Negotiation successful
0807 02:26:36.900561 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":183,"threads_started":1}
I20260807 02:26:36.900772 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.890641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43228 (local address 127.4.221.126:39367)
0807 02:26:36.890811 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:36.890816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:36.891225 (+ 409us) server_negotiation.cc:407] Connection header received
0807 02:26:36.891421 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:36.891423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:36.891469 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:36.891536 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:36.892501 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.893347 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.894455 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.894674 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.894784 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.897091 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:36.897094 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:36.897096 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:36.897120 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:36.898745 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.899417 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.899420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.899421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.899512 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.899804 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.899806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.899807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.899990 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:36.900102 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:36.900447 (+ 345us) server_negotiation.cc:299] Negotiation successful
0807 02:26:36.900560 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 02:26:36.902113 28722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.126@KRBTEST.COM
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.124@KRBTEST.COM for kudu/127.4.221.125@KRBTEST.COM
I20260807 02:26:36.911962 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.901575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43242 (local address 127.4.221.126:39367)
0807 02:26:36.901677 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:36.901688 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:36.901700 (+ 12us) server_negotiation.cc:407] Connection header received
0807 02:26:36.901733 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:36.901735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:36.901775 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:36.901884 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:36.902587 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.903313 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.904394 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.904731 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.904848 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.908422 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:36.908429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:36.908436 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:36.908466 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:36.910254 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.910691 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.910694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.910696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.910790 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.911068 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.911072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.911073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.911271 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:36.911517 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:36.911648 (+ 131us) server_negotiation.cc:299] Negotiation successful
0807 02:26:36.911777 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260807 02:26:36.913055 28570 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } attempt: 3
I20260807 02:26:36.915789 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.902621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.125:45121 (local address 127.0.0.1:46996)
0807 02:26:36.902748 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.902764 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:36.902834 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:36.903210 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:36.903214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:36.903385 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:36.903480 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.903485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.904491 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.904495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.905251 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.905263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.905951 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.905954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.906033 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.906504 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:36.906523 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:36.908317 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:36.913771 (+ 5454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.913776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.913778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.914091 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.914476 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.914479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.914483 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.914577 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.915144 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:36.915149 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:36.915301 (+ 152us) client_negotiation.cc:769] Sending connection context
0807 02:26:36.915439 (+ 138us) client_negotiation.cc:240] Negotiation successful
0807 02:26:36.915594 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260807 02:26:36.915868 28651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.902794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.4.221.125:45121)
0807 02:26:36.902946 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:36.902950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:36.902964 (+ 14us) server_negotiation.cc:407] Connection header received
0807 02:26:36.903005 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:36.903008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:36.903049 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:36.903130 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:36.903600 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.904374 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.905389 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.905792 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.905963 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.908486 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:36.908490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:36.908494 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:36.908526 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:36.913627 (+ 5101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.914243 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.914248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.914251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.914351 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.914704 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.914708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.914710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.915002 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:36.915134 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:36.915537 (+ 403us) server_negotiation.cc:299] Negotiation successful
0807 02:26:36.915688 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260807 02:26:36.916742 28645 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } attempt: 3
I20260807 02:26:36.918241 28722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } has no permanent_uuid. Determining permanent_uuid...
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.126@KRBTEST.COM for kudu/127.4.221.124@KRBTEST.COM
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.221.125@KRBTEST.COM
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.125@KRBTEST.COM for kudu/127.4.221.124@KRBTEST.COM
Aug 07 02:26:36 dist-test-slave-69rg krb5kdc[28488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786069596, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.221.124@KRBTEST.COM for kudu/127.4.221.124@KRBTEST.COM
I20260807 02:26:36.926934 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.913453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:43005 (local address 127.0.0.1:43534)
0807 02:26:36.913618 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.913632 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:36.913710 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:36.916133 (+ 2423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:36.916139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:36.916347 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:36.916477 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.916487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.918544 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.918548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.919538 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.919551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.920170 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.920174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.920262 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.921201 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:36.921217 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:36.923009 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:36.924988 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.924994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.924999 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.925269 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.925604 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.925610 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.925612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.925719 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.926266 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:36.926272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:36.926403 (+ 131us) client_negotiation.cc:769] Sending connection context
0807 02:26:36.926526 (+ 123us) client_negotiation.cc:240] Negotiation successful
0807 02:26:36.926709 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260807 02:26:36.927479 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.914077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43534 (local address 127.4.221.124:43005)
0807 02:26:36.915901 (+ 1824us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:36.915906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:36.915920 (+ 14us) server_negotiation.cc:407] Connection header received
0807 02:26:36.915957 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:36.915960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:36.916001 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:36.916603 (+ 602us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:36.916722 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.918402 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.919709 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.920032 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.920491 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.923178 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:36.923184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:36.923191 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:36.923220 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:36.924865 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.925398 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.925402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.925404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.925490 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.925879 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.925883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.925886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.926147 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:36.926255 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:36.927173 (+ 918us) server_negotiation.cc:299] Negotiation successful
0807 02:26:36.927292 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1725}
I20260807 02:26:36.929541 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.917154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:43005 (local address 127.0.0.1:43544)
0807 02:26:36.917455 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.917470 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:36.917541 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:36.917765 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:36.917768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:36.917976 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:36.918063 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.918068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.919811 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.919815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.920586 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.920601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.921341 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.921344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.921438 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.922570 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:36.922586 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:36.925254 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:36.927778 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.927783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.927787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.928119 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.928455 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.928460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.928463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.928562 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.929005 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:36.929008 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:36.929125 (+ 117us) client_negotiation.cc:769] Sending connection context
0807 02:26:36.929215 (+ 90us) client_negotiation.cc:240] Negotiation successful
0807 02:26:36.929376 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232}
I20260807 02:26:36.930146 28727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.917213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.4.221.124:43005)
0807 02:26:36.917507 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:36.917514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:36.917526 (+ 12us) server_negotiation.cc:407] Connection header received
0807 02:26:36.917615 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:36.917617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:36.917648 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:36.917719 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:36.918257 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.919670 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.920752 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.921194 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.921651 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.925401 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:36.925404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:36.925407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:36.925455 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:36.927654 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.928249 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.928252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.928255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.928342 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.928691 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.928694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.928696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.928894 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:36.929733 (+ 839us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:36.929798 (+ 65us) server_negotiation.cc:299] Negotiation successful
0807 02:26:36.929963 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"spinlock_wait_cycles":3840,"thread_start_us":97,"threads_started":1}
I20260807 02:26:36.930367 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.918755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.221.124:43005 (local address 127.0.0.1:43548)
0807 02:26:36.918926 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0807 02:26:36.918942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0807 02:26:36.919016 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0807 02:26:36.919971 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0807 02:26:36.919974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0807 02:26:36.920159 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0807 02:26:36.920252 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.920259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.921314 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.921318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.922246 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0807 02:26:36.922261 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.923170 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.923175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0807 02:26:36.923250 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.923810 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0807 02:26:36.923829 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0807 02:26:36.925903 (+ 2074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0807 02:26:36.928596 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.928601 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.928605 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.928831 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.929126 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0807 02:26:36.929129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0807 02:26:36.929131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0807 02:26:36.929219 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0807 02:26:36.929641 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0807 02:26:36.929644 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0807 02:26:36.929733 (+ 89us) client_negotiation.cc:769] Sending connection context
0807 02:26:36.929877 (+ 144us) client_negotiation.cc:240] Negotiation successful
0807 02:26:36.930229 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":228}
I20260807 02:26:36.930503 28728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.918805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.4.221.124:43005)
0807 02:26:36.919658 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:36.919676 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:36.919690 (+ 14us) server_negotiation.cc:407] Connection header received
0807 02:26:36.919720 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:36.919723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:36.919762 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:36.919844 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:36.920365 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.921186 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.922438 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.922948 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.923055 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.926042 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:36.926045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:36.926047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:36.926084 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:36.928495 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.928931 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.928937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.928942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.929012 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.929305 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.929307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.929309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.929486 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:36.929578 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:36.930001 (+ 423us) server_negotiation.cc:299] Negotiation successful
0807 02:26:36.930087 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":792,"spinlock_wait_cycles":23808,"thread_start_us":152,"threads_started":1}
I20260807 02:26:36.931960 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.914368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.4.221.125:45121)
0807 02:26:36.915538 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:36.915544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:36.915559 (+ 15us) server_negotiation.cc:407] Connection header received
0807 02:26:36.915596 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:36.915599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:36.915643 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:36.915735 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:36.916948 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.917664 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.919542 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.920077 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.920241 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.926512 (+ 6271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:36.926516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:36.926520 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:36.926550 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:36.929188 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.929900 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.929903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.929905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.930004 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.930827 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.930830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.930831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.931040 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:36.931264 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:36.931704 (+ 440us) server_negotiation.cc:299] Negotiation successful
0807 02:26:36.931825 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1084}
I20260807 02:26:36.932379 28570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35: Bootstrap starting.
I20260807 02:26:36.933015 28645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d: Bootstrap starting.
I20260807 02:26:36.935434 28645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:36.935467 28570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:36.936048 4981 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260807 02:26:36.936265 28645 log.cc:824] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:36.936295 28570 log.cc:824] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:36.938482 28645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d: No bootstrap required, opened a new log
I20260807 02:26:36.937458 28543 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "34d2236078c249948987f746fa8aba35"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43242
I20260807 02:26:36.939524 28570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35: No bootstrap required, opened a new log
I20260807 02:26:36.941286 28722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a: Bootstrap starting.
I20260807 02:26:36.942466 28645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } }
I20260807 02:26:36.942467 28570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } }
I20260807 02:26:36.942708 28570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d2236078c249948987f746fa8aba35, State: Initialized, Role: FOLLOWER
I20260807 02:26:36.942699 28645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3800131fd11345dc9e4cced913cd807d, State: Initialized, Role: FOLLOWER
I20260807 02:26:36.943051 28570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } }
I20260807 02:26:36.943051 28645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } }
I20260807 02:26:36.943945 28722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a: Neither blocks nor log segments found. Creating new log.
I20260807 02:26:36.943986 28732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } } }
I20260807 02:26:36.944425 28645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:36.944936 28722 log.cc:824] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a: Log is configured to *not* fsync() on all Append() calls
I20260807 02:26:36.944993 28731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } } }
I20260807 02:26:36.945135 28731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:26:36.944425 28570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:36.944512 28543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260807 02:26:36.945432 28543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:36.945269 28732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:26:36.946889 28543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } }
I20260807 02:26:36.947331 28722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a: No bootstrap required, opened a new log
I20260807 02:26:36.947938 28543 leader_election.cc:288] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [CANDIDATE]: Term 1 election: Requested vote from peers 3800131fd11345dc9e4cced913cd807d (127.4.221.125:45121), 4acf9c9c9bda435daeee0e7792d3e18a (127.4.221.124:43005)
I20260807 02:26:36.948561 28619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34d2236078c249948987f746fa8aba35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3800131fd11345dc9e4cced913cd807d"
I20260807 02:26:36.949518 28619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:36.950847 28619 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34d2236078c249948987f746fa8aba35 in term 1.
I20260807 02:26:36.951372 28520 leader_election.cc:302] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34d2236078c249948987f746fa8aba35, 3800131fd11345dc9e4cced913cd807d; no voters:
I20260807 02:26:36.951835 28732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [term 1 FOLLOWER]: Leader election won for term 1
I20260807 02:26:36.952076 28732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [term 1 LEADER]: Becoming Leader. State: Replica: 34d2236078c249948987f746fa8aba35, State: Running, Role: LEADER
I20260807 02:26:36.952395 28732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } }
I20260807 02:26:36.952600 28722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } }
I20260807 02:26:36.952822 28722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acf9c9c9bda435daeee0e7792d3e18a, State: Initialized, Role: FOLLOWER
I20260807 02:26:36.953141 28722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } }
I20260807 02:26:36.954262 28732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34d2236078c249948987f746fa8aba35. Latest consensus state: current_term: 1 leader_uuid: "34d2236078c249948987f746fa8aba35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } } }
I20260807 02:26:36.954357 28732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:36.954623 28722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a [sys.catalog]: configured and running, proceeding with master startup.
I20260807 02:26:36.954813 28696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34d2236078c249948987f746fa8aba35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4acf9c9c9bda435daeee0e7792d3e18a"
I20260807 02:26:36.955029 28696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a [term 0 FOLLOWER]: Advancing to term 1
I20260807 02:26:36.956485 28696 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34d2236078c249948987f746fa8aba35 in term 1.
W20260807 02:26:36.956506 28754 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:26:36.956574 28754 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:26:36.957178 28763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260807 02:26:36.954506 28501 master_runner.cc:186] Error getting master registration for 127.4.221.124:43005: OK, instance_id { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" instance_seqno: 1786069596877266 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260807 02:26:36.958530 28501 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 02:26:36.960156 28763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260807 02:26:36.961962 28761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } } }
I20260807 02:26:36.962028 28577 master_runner.cc:263] No leader master found from master 3800131fd11345dc9e4cced913cd807d
I20260807 02:26:36.962098 28577 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260807 02:26:36.962105 28761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a [sys.catalog]: This master's current role is: FOLLOWER
W20260807 02:26:36.963018 28775 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260807 02:26:36.963332 28775 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260807 02:26:36.973763 28696 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a [term 1 FOLLOWER]: Refusing update from remote peer 34d2236078c249948987f746fa8aba35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:26:36.974617 28760 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:26:36.975360 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.964394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43254 (local address 127.4.221.126:39367)
0807 02:26:36.964552 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:36.964557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:36.965556 (+ 999us) server_negotiation.cc:407] Connection header received
0807 02:26:36.965768 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:36.965772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:36.965850 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:36.965908 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:36.967049 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.967918 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.969148 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.969487 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.969603 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.971542 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:36.971546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:36.971549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:36.971578 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:36.973356 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.973924 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.973928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.973929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.974021 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.974358 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.974361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.974363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.974580 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:36.974702 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:36.975066 (+ 364us) server_negotiation.cc:299] Negotiation successful
0807 02:26:36.975202 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260807 02:26:36.979660 28761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34d2236078c249948987f746fa8aba35. Latest consensus state: current_term: 1 leader_uuid: "34d2236078c249948987f746fa8aba35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } } }
I20260807 02:26:36.979822 28761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:26:36.981007 28619 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d [term 1 FOLLOWER]: Refusing update from remote peer 34d2236078c249948987f746fa8aba35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260807 02:26:36.981662 28760 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260807 02:26:36.984937 28728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.964915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.4.221.124:43005)
0807 02:26:36.965025 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:36.965029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:36.970105 (+ 5076us) server_negotiation.cc:407] Connection header received
0807 02:26:36.970299 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:36.970302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:36.970348 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:36.970397 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:36.971636 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.972334 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.974464 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.974821 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.974901 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.978832 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:36.978836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:36.978839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:36.978865 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:36.980768 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.982384 (+ 1616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.982387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.982389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.982484 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.983905 (+ 1421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.983908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.983910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.984147 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:36.984267 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:36.984647 (+ 380us) server_negotiation.cc:299] Negotiation successful
0807 02:26:36.984776 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260807 02:26:36.985992 28731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34d2236078c249948987f746fa8aba35. Latest consensus state: current_term: 1 leader_uuid: "34d2236078c249948987f746fa8aba35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } } }
I20260807 02:26:36.986125 28731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:26:36.990331 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:36.965601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.4.221.125:45121)
0807 02:26:36.965711 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:36.965715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:36.970843 (+ 5128us) server_negotiation.cc:407] Connection header received
0807 02:26:36.973444 (+ 2601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:36.973448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:36.973501 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:36.973555 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:36.976761 (+ 3206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.977523 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.981298 (+ 3775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:36.981616 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:36.981702 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:36.986577 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0807 02:26:36.986581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0807 02:26:36.986584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0807 02:26:36.986614 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0807 02:26:36.988387 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.988907 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.988911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.988912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.988996 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0807 02:26:36.989320 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0807 02:26:36.989324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0807 02:26:36.989326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0807 02:26:36.989540 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0807 02:26:36.989655 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:36.990048 (+ 393us) server_negotiation.cc:299] Negotiation successful
0807 02:26:36.990180 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260807 02:26:36.994922 28760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34d2236078c249948987f746fa8aba35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } } }
I20260807 02:26:36.995055 28760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:36.996411 28761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34d2236078c249948987f746fa8aba35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } } }
I20260807 02:26:36.996519 28761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:26:36.996980 28778 mvcc.cc:205] Tried to move back new op lower bound from 7315741069196439552 to 7315741069120557056. Current Snapshot: MvccSnapshot[applied={T|T < 7315741069196439552}]
I20260807 02:26:36.997742 28732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "34d2236078c249948987f746fa8aba35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } } }
I20260807 02:26:36.997902 28732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35 [sys.catalog]: This master's current role is: LEADER
I20260807 02:26:36.998890 28731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "34d2236078c249948987f746fa8aba35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "34d2236078c249948987f746fa8aba35" member_type: VOTER last_known_addr { host: "127.4.221.126" port: 39367 } } peers { permanent_uuid: "3800131fd11345dc9e4cced913cd807d" member_type: VOTER last_known_addr { host: "127.4.221.125" port: 45121 } } peers { permanent_uuid: "4acf9c9c9bda435daeee0e7792d3e18a" member_type: VOTER last_known_addr { host: "127.4.221.124" port: 43005 } } }
I20260807 02:26:36.999009 28731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d [sys.catalog]: This master's current role is: FOLLOWER
I20260807 02:26:36.999148 28782 mvcc.cc:205] Tried to move back new op lower bound from 7315741069196439552 to 7315741069120557056. Current Snapshot: MvccSnapshot[applied={T|T < 7315741069196439552}]
I20260807 02:26:37.001641 28784 mvcc.cc:205] Tried to move back new op lower bound from 7315741069196439552 to 7315741069120557056. Current Snapshot: MvccSnapshot[applied={T|T < 7315741069196439552}]
I20260807 02:26:37.006697 28763 catalog_manager.cc:1347] Generated new cluster ID: 661b59ef919945548a1c9890bd8026f3
I20260807 02:26:37.006776 28763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260807 02:26:37.037989 28763 catalog_manager.cc:1370] Generated new certificate authority record
I20260807 02:26:37.039299 28763 catalog_manager.cc:1504] Loading token signing keys...
I20260807 02:26:37.055166 28763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34d2236078c249948987f746fa8aba35: Generated new TSK 0
I20260807 02:26:37.059295 28763 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260807 02:26:37.097783 28650 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:37.094198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47024 (local address 127.4.221.125:45121)
0807 02:26:37.094328 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:37.094333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:37.096558 (+ 2225us) server_negotiation.cc:407] Connection header received
0807 02:26:37.096776 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:37.096780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:37.096834 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:37.096899 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:37.097636 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47024: BlockingRecv error: recv got EOF from 127.0.0.1:47024 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260807 02:26:37.099900 28728 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:37.095218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.4.221.124:43005)
0807 02:26:37.095346 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:37.095350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:37.098264 (+ 2914us) server_negotiation.cc:407] Connection header received
0807 02:26:37.098863 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:37.098867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:37.098918 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:37.098970 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:37.099762 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43576: BlockingRecv error: recv got EOF from 127.0.0.1:43576 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260807 02:26:37.100633 28574 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:37.095945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.4.221.126:39367)
0807 02:26:37.096063 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:37.096067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:37.098590 (+ 2523us) server_negotiation.cc:407] Connection header received
0807 02:26:37.099104 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:37.099107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:37.099177 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:37.099230 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:37.100518 (+ 1288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43268: BlockingRecv error: recv got EOF from 127.0.0.1:43268 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260807 02:26:37.105386 28650 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:37.103729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.4.221.125:45121)
0807 02:26:37.103837 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:37.103841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:37.104414 (+ 573us) server_negotiation.cc:407] Connection header received
0807 02:26:37.104632 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:37.104635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:37.104710 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:37.104773 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:37.105286 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47026: BlockingRecv error: recv got EOF from 127.0.0.1:47026 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260807 02:26:37.110302 28650 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:37.108689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.4.221.125:45121)
0807 02:26:37.108786 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:37.108790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:37.109303 (+ 513us) server_negotiation.cc:407] Connection header received
0807 02:26:37.109481 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:37.109484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:37.109544 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:37.109594 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:37.110202 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47042: BlockingRecv error: recv got EOF from 127.0.0.1:47042 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260807 02:26:37.115705 28650 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:37.114240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.4.221.125:45121)
0807 02:26:37.114339 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:37.114343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:37.114846 (+ 503us) server_negotiation.cc:407] Connection header received
0807 02:26:37.115024 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:37.115027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:37.115074 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:37.115134 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:37.115606 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47044: BlockingRecv error: recv got EOF from 127.0.0.1:47044 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260807 02:26:37.123499 28650 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:37.121730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.4.221.125:45121)
0807 02:26:37.121907 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:37.121911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:37.122472 (+ 561us) server_negotiation.cc:407] Connection header received
0807 02:26:37.122662 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:37.122665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:37.122719 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:37.122770 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:37.123390 (+ 620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47050: BlockingRecv error: recv got EOF from 127.0.0.1:47050 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260807 02:26:37.135474 28650 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:37.133727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.4.221.125:45121)
0807 02:26:37.133872 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:37.133876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:37.134454 (+ 578us) server_negotiation.cc:407] Connection header received
0807 02:26:37.134656 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:37.134659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:37.134717 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:37.134768 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:37.135358 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47054: BlockingRecv error: recv got EOF from 127.0.0.1:47054 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260807 02:26:37.157385 28650 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:37.155372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47066 (local address 127.4.221.125:45121)
0807 02:26:37.155503 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:37.155508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:37.156273 (+ 765us) server_negotiation.cc:407] Connection header received
0807 02:26:37.156513 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:37.156517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:37.156577 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:37.156639 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:37.157262 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47066: BlockingRecv error: recv got EOF from 127.0.0.1:47066 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260807 02:26:37.193276 28650 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:37.192119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47070 (local address 127.4.221.125:45121)
0807 02:26:37.192302 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:37.192306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:37.192380 (+ 74us) server_negotiation.cc:407] Connection header received
0807 02:26:37.192499 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:37.192501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:37.192549 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:37.192627 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:37.193092 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47070: BlockingRecv error: recv got EOF from 127.0.0.1:47070 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260807 02:26:37.260505 28650 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:37.259457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.4.221.125:45121)
0807 02:26:37.259612 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:37.259616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:37.259771 (+ 155us) server_negotiation.cc:407] Connection header received
0807 02:26:37.259843 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:37.259845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:37.259900 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:37.259989 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:37.260405 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47082: BlockingRecv error: recv got EOF from 127.0.0.1:47082 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260807 02:26:37.392092 28650 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:37.390813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.4.221.125:45121)
0807 02:26:37.390992 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:37.390997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:37.391121 (+ 124us) server_negotiation.cc:407] Connection header received
0807 02:26:37.391330 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:37.391334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:37.391393 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:37.391489 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:37.391969 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47090: BlockingRecv error: recv got EOF from 127.0.0.1:47090 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260807 02:26:37.651530 28650 negotiation.cc:337] Failed RPC negotiation. Trace:
0807 02:26:37.650367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47094 (local address 127.4.221.125:45121)
0807 02:26:37.650532 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:37.650537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:37.650554 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:26:37.650723 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:37.650727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:37.650786 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:37.650881 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0807 02:26:37.651406 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47094: BlockingRecv error: recv got EOF from 127.0.0.1:47094 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260807 02:26:37.958048 28754 catalog_manager.cc:1259] Loaded cluster ID: 661b59ef919945548a1c9890bd8026f3
I20260807 02:26:37.958204 28754 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d: loading cluster ID for follower catalog manager: success
I20260807 02:26:37.959857 28754 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d: acquiring CA information for follower catalog manager: success
I20260807 02:26:37.960966 28754 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3800131fd11345dc9e4cced913cd807d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 02:26:37.964421 28775 catalog_manager.cc:1259] Loaded cluster ID: 661b59ef919945548a1c9890bd8026f3
I20260807 02:26:37.964478 28775 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a: loading cluster ID for follower catalog manager: success
I20260807 02:26:37.965963 28775 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a: acquiring CA information for follower catalog manager: success
I20260807 02:26:37.966941 28775 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4acf9c9c9bda435daeee0e7792d3e18a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260807 02:26:38.170954 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:38.166225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47110 (local address 127.4.221.125:45121)
0807 02:26:38.166538 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:38.166543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:38.166658 (+ 115us) server_negotiation.cc:407] Connection header received
0807 02:26:38.166758 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:38.166760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:38.166934 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:38.167055 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:26:38.167312 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:38.167900 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:38.168974 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:38.169248 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:38.169348 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:38.169749 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:26:38.170400 (+ 651us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:26:38.170516 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:38.170639 (+ 123us) server_negotiation.cc:299] Negotiation successful
0807 02:26:38.170705 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260807 02:26:38.178843 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:38.174755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43588 (local address 127.4.221.124:43005)
0807 02:26:38.175041 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:38.175046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:38.175063 (+ 17us) server_negotiation.cc:407] Connection header received
0807 02:26:38.175205 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:38.175210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:38.175363 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:38.175517 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:26:38.175739 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:38.176301 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:38.177167 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:38.177415 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:38.177532 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:38.177884 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:26:38.178403 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:26:38.178554 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:38.178592 (+ 38us) server_negotiation.cc:299] Negotiation successful
0807 02:26:38.178655 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":82,"threads_started":1}
I20260807 02:26:38.186694 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:38.181992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.4.221.125:45121)
0807 02:26:38.182136 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:38.182142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:38.182651 (+ 509us) server_negotiation.cc:407] Connection header received
0807 02:26:38.182843 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:38.182846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:38.182904 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:38.183013 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:26:38.183413 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:38.184203 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:38.185290 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:38.185614 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:38.185740 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:38.186041 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:26:38.186160 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:26:38.186252 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:38.186384 (+ 132us) server_negotiation.cc:299] Negotiation successful
0807 02:26:38.186455 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260807 02:26:38.187273 28862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:38.182173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43590 (local address 127.4.221.124:43005)
0807 02:26:38.182304 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:38.182312 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:38.182364 (+ 52us) server_negotiation.cc:407] Connection header received
0807 02:26:38.182560 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:38.182564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:38.182640 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:38.182691 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:26:38.183317 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:38.184127 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:38.185580 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:38.186026 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:38.186148 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:38.186649 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:26:38.186749 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:26:38.186810 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:38.187044 (+ 234us) server_negotiation.cc:299] Negotiation successful
0807 02:26:38.187112 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260807 02:26:38.187732 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0807 02:26:38.181990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.4.221.126:39367)
0807 02:26:38.182598 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0807 02:26:38.182604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0807 02:26:38.182620 (+ 16us) server_negotiation.cc:407] Connection header received
0807 02:26:38.182738 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0807 02:26:38.182741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0807 02:26:38.182963 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0807 02:26:38.183088 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0807 02:26:38.183541 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:38.184165 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:38.185601 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0807 02:26:38.186008 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0807 02:26:38.186103 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0807 02:26:38.186477 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0807 02:26:38.186912 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0807 02:26:38.187007 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0807 02:26:38.187429 (+ 422us) server_negotiation.cc:299] Negotiation successful
0807 02:26:38.187502 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":76,"threads_started":1}
I20260807 02:26:38.196729 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 28501
I20260807 02:26:38.207662 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 28577
I20260807 02:26:38.218492 4981 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskBkVddJ/build/debug/bin/kudu with pid 28653
2026-08-07T02:26:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3263 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6625 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (296373 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST